Oct 31 18:20:12.017348 ip-10-0-142-120 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 18:20:12.354087 ip-10-0-142-120 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 18:20:12.354087 ip-10-0-142-120 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 18:20:12.354087 ip-10-0-142-120 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 18:20:12.354087 ip-10-0-142-120 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 18:20:12.357814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357737    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:12.357814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357815    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357819    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357822    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357825    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357827    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357830    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357832    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357835    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357837    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357840    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357842    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357844    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357847    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357850    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357852    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357854    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357857    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357859    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357862    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:12.357880 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357864    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357866    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357869    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357871    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357873    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357876    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357878    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357881    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357889    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357892    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357895    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357897    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357900    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357902    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357905    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357907    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357909    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357912    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357916    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357920    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:12.358369 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357923    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357926    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357929    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357932    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357934    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357937    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357939    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357941    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357944    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357946    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357949    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357951    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357954    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357957    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357960    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357962    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357965    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357967    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357970    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357972    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:12.358814 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357974    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357977    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357979    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357981    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357984    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357988    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357991    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357993    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357996    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.357998    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358001    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358003    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358006    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358008    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358011    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358014    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358016    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358018    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358021    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:12.359291 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358024    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358026    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358029    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358031    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358033    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358035    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358038    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358041    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358045    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358061    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358064    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358067    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358071    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358074    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358076    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358079    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358081    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358084    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358086    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358088    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:12.359720 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358091    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358093    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358096    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358098    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358101    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358103    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358105    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358829    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358835    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358838    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358850    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358853    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358857    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358860    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358863    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358865    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358868    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358871    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358874    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358876    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:12.360215 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358879    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358881    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358885    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358888    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358891    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358894    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358896    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358899    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358901    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358903    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358906    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358908    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358911    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358913    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358915    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358918    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358920    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358923    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358925    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358928    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:12.360695 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358930    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358933    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358935    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358937    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358940    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358943    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358946    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358949    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358951    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358953    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358956    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358959    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358961    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358964    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358966    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358968    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358971    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358973    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358975    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358978    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:12.361170 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358980    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358982    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358985    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358987    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358990    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358992    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358994    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.358997    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359000    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359002    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359004    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359007    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359009    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359012    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359014    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359016    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359019    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359021    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359023    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359026    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:12.362040 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359028    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359031    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359033    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359035    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359038    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359040    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359043    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359045    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359066    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359070    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359077    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359082    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359086    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359089    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359091    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359094    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359096    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359099    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:12.362504 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359101    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359104    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359106    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359109    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359112    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359114    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359116    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359119    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359121    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359123    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359126    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359128    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359130    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359134    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359137    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359684    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359689    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359692    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359695    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:12.362949 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359698    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359700    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359703    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359705    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359708    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359710    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359713    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359715    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359717    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359720    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359723    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359725    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359728    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359731    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359733    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359736    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359738    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359741    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359743    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:12.363401 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359746    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359748    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359750    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359753    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359755    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359758    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359760    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359768    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359770    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359775    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359778    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359780    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359783    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359785    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359787    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359790    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359792    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359794    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359797    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:12.363832 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359799    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359802    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359804    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359806    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359809    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359811    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359813    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359816    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359819    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359821    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359823    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359827    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359831    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359834    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359837    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359840    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359842    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359844    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359847    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:12.364319 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359850    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359852    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359861    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359864    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359867    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359869    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359872    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359874    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359877    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359879    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359882    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359884    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359888    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359892    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359896    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359899    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359901    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359904    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359906    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359910    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:12.364753 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359914    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359917    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359919    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359921    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359924    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359926    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359929    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359931    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359933    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359936    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359938    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359941    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359943    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359945    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359948    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359957    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359960    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359964    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359966    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:12.365272 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359969    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359971    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359973    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359976    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359978    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.359981    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361203    2573 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361212    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361224    2573 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361229    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361234    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361237    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361241    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361246    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361249    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361253    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361256    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361259    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361262    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361265    2573 flags.go:64] FLAG: --cgroup-root=""
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361267    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361270    2573 flags.go:64] FLAG: --client-ca-file=""
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361273    2573 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 18:20:12.365732 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361275    2573 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361283    2573 flags.go:64] FLAG: --cluster-domain=""
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361285    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361288    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361291    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361296    2573 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361306    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361309    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361312    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361315    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361318    2573 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361320    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361323    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361326    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361329    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361333    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361336    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361339    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361341    2573 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361344    2573 flags.go:64] FLAG: --enable-server="true"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361347    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361353    2573 flags.go:64] FLAG: --event-burst="100"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361357    2573 flags.go:64] FLAG: --event-qps="50"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361361    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 18:20:12.366263 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361364    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361367    2573 flags.go:64] FLAG: --eviction-hard=""
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361371    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361374    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361377    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361380    2573 flags.go:64] FLAG: --eviction-soft=""
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361382    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361385    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361388    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361391    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361394    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361396    2573 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361399    2573 flags.go:64] FLAG: --feature-gates=""
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361403    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361406    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361414    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361417    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361420    2573 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361423    2573 flags.go:64] FLAG: --help="false"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361426    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361429    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361432    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361434    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 18:20:12.366832 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361438    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361441    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361444    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361447    2573 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361449    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361452    2573 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361455    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361459    2573 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361462    2573 flags.go:64] FLAG: --kube-reserved=""
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361465    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361467    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361470    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361473    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361475    2573 flags.go:64] FLAG: --lock-file=""
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361478    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361481    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361484    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361489    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361492    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361494    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361497    2573 flags.go:64] FLAG: --logging-format="text"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361500    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361503    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361506    2573 flags.go:64] FLAG: --manifest-url=""
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361508    2573 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 18:20:12.367380 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361521    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361524    2573 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361528    2573 flags.go:64] FLAG: --max-pods="110"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361531    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361533    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361536    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361539    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361542    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361545    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361548    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361554    2573 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361557    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361560    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361563    2573 flags.go:64] FLAG: --pod-cidr=""
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361567    2573 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361570    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361572    2573 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361575    2573 flags.go:64] FLAG: --port="10250"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361578    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361581    2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0541ad9bd068e0419"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361584    2573 flags.go:64] FLAG: --qos-reserved=""
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361586    2573 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361589    2573 flags.go:64] FLAG: --register-node="true"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361592    2573 flags.go:64] FLAG: --register-with-taints=""
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361596    2573 flags.go:64] FLAG: --registry-burst="10"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361599    2573 flags.go:64] FLAG: --registry-qps="5"
Oct 31 18:20:12.367958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361601    2573 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361604    2573 flags.go:64] FLAG: --reserved-memory=""
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361608    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361610    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361613    2573 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361616    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361619    2573 flags.go:64] FLAG: --runonce="false"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361627    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361630    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361633    2573 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361635    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361638    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361641    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361644    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361647    2573 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361649    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361652    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361655    2573 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361658    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361661    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361664    2573 flags.go:64] FLAG: --system-cgroups=""
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361667    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361673    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361675    2573 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361678    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 18:20:12.368626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361684    2573 flags.go:64] FLAG: --tls-min-version=""
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361687    2573 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361690    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361693    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361695    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361698    2573 flags.go:64] FLAG: --v="2"
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361702    2573 flags.go:64] FLAG: --version="false"
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361706    2573 flags.go:64] FLAG: --vmodule=""
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361710    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.361714    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361898    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361901    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361904    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361906    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361909    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361917    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361919    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361922    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361924    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361927    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361929    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:12.369232 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361931    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361934    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361936    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361939    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361941    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361943    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361948    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361950    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361953    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361955    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361958    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361961    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361963    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361966    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361968    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361971    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361973    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361975    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361978    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361980    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:12.369694 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361984    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361986    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361989    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361991    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361993    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361996    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.361999    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362006    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362009    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362011    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362014    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362016    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362018    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362021    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362023    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362027    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362030    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362033    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362037    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:12.370158 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362040    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362042    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362045    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362059    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362062    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362065    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362068    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362070    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362073    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362076    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362080    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362083    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362086    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362090    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362093    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362095    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362098    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362100    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362103    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362105    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:12.370587 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362108    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362115    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362118    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362121    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362123    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362125    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362128    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362130    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362132    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362134    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362137    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362140    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362142    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362144    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362147    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362149    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362151    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362154    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362156    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362158    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:12.371059 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362161    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362163    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362165    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362168    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362170    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362174    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362176    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362179    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362181    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362184    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362186    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362188    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362191    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362193    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362201    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362203    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:12.371521 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.362229    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362265    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362269    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362272    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362274    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362277    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362279    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362282    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362284    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362287    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362289    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362292    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362294    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362296    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362299    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362301    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362304    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362306    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362308    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:12.371893 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362310    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362313    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362317    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362319    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362322    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362324    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362326    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362329    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362331    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362333    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362336    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362338    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362341    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362343    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362345    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362348    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362350    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362353    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362355    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:12.372377 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362357    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362360    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362362    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362364    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362366    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362369    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362371    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362374    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362377    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362381    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362384    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362388    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362390    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362393    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362395    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362399    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362401    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362403    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362406    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:12.372797 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362408    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362410    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362414    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362417    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362420    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362422    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362425    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362428    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362430    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362432    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362435    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362437    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362439    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362442    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362444    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362446    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362449    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362451    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362453    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362456    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:12.373242 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362458    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362460    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362462    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362465    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362468    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362470    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362472    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362475    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362479    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362481    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362483    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362486    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362488    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362490    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362494    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362497    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362499    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362501    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362504    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362507    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:12.373680 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362510    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362512    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362515    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362517    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362520    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362522    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362525    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362527    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362530    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.362533    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.369021    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.369037    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.369068    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.369074    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369131    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369137    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369139    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369142    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369145    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:12.374178 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369147    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369150    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369153    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369155    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369158    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369160    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369163    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369166    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369168    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369171    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369173    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369175    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369178    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369180    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369182    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369184    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369187    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369189    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369192    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:12.374605 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369194    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369197    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369199    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369202    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369204    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369209    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369211    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369214    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369216    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369219    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369221    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369223    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369226    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369228    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369230    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369233    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369235    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369238    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369241    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:12.375066 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369243    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369245    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369248    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369250    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369252    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369255    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369257    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369260    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369263    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369265    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369269    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369272    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369275    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369316    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369319    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369321    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369324    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369326    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369329    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369332    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:12.375514 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369335    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369338    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369340    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369342    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369345    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369347    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369350    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369352    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369354    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369356    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369359    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369361    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369364    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369366    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369368    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369371    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369373    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369375    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369378    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369380    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:12.375957 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369383    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369385    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369388    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369390    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369392    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369395    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369397    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369399    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369402    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369404    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369406    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369409    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369412    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369414    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369416    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369419    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369422    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369426    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369429    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:12.376464 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369433    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369436    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369439    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369441    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.369468    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369601    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369606    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369608    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369611    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369613    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369615    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369618    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369620    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369623    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369625    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:12.376883 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369629    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369632    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369634    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369636    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369638    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369641    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369643    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369645    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369648    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369650    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369652    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369656    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369658    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369661    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369663    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369665    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369667    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369670    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369673    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369675    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:12.377284 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369677    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369681    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369685    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369687    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369690    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369692    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369694    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369697    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369699    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369702    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369705    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369707    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369709    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369712    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369714    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369716    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369719    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369721    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369723    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:12.377909 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369726    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369728    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369730    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369733    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369735    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369738    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369741    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369743    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369745    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369748    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369751    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369755    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369758    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369760    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369763    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369765    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369767    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369770    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369772    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369774    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:12.378627 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369777    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369779    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369782    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369784    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369787    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369789    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369792    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369794    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369796    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369798    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369801    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369803    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369806    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369808    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369810    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369812    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369815    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369817    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369820    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:12.379120 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369822    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369824    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369827    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369829    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369832    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369834    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369836    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369838    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369841    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369843    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369845    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369848    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369850    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369852    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369855    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369861    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369863    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:12.379549 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369865    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.369888    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369919    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369922    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369925    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369928    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369930    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369932    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369935    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369937    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369940    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369942    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369945    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369947    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369949    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369952    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:12.379941 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369955    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369957    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369959    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369962    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369964    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369966    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369968    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369971    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369974    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369976    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369978    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369981    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369983    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369986    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369988    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369992    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369995    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.369998    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370000    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:12.380339 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370003    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370005    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370008    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370010    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370012    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370015    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370017    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370019    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370022    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370024    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370026    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370029    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370031    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370034    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370036    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370039    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370043    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370046    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370065    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370068    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:12.380767 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370070    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370073    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370075    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370078    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370080    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370083    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370085    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370087    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370091    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370097    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370099    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370102    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370104    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370107    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370109    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370112    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370115    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370117    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370119    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:12.381390 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370122    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370124    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370127    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370129    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370133    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370137    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370139    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370143    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370145    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370148    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370150    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370152    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370155    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370157    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370159    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370162    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370165    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370167    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370170    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:12.381939 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370172    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370175    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370178    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370182    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370184    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370187    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370189    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370191    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370194    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370196    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370198    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370201    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370203    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370207    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:12.370209    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.370258    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.372087    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.372903    2573 server.go:1014] "Starting client certificate rotation"
Oct 31 18:20:12.382391 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.372994    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 18:20:12.382793 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.373498    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 18:20:12.382793 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.374368    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 18:20:12.382793 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.379780    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 18:20:12.391606 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.391584    2573 log.go:25] "Validated CRI v1 runtime API"
Oct 31 18:20:12.394296 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.394284    2573 log.go:25] "Validated CRI v1 image API"
Oct 31 18:20:12.395117 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.395096    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 18:20:12.397195 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.397178    2573 fs.go:135] Filesystem UUIDs: map[075e0292-ef41-4f6c-a261-31d551eb0bcb:/dev/nvme0n1p3 21374453-50fe-4038-970c-69c026217ad8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 31 18:20:12.397238 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.397196    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 18:20:12.402157 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.402044    2573 manager.go:217] Machine: {Timestamp:2026-10-31 18:20:12.400715106 +0000 UTC m=+0.277255400 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200011 MemoryCapacity:32812023808 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28740c198459331d15e8d3987a2e2d SystemUUID:ec28740c-1984-5933-1d15-e8d3987a2e2d BootID:b6a2d5a4-2f20-44de-8494-a56a96be2979 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406011904 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:44:ed:96:a0:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:44:ed:96:a0:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:60:11:d6:c2:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812023808 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 18:20:12.402626 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.402617    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 18:20:12.402734 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.402723    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 18:20:12.404840 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.404816    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 18:20:12.404971 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.404843    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-120.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 18:20:12.405015 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.404974    2573 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 18:20:12.405015 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.404982    2573 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 18:20:12.405080 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.405015    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 18:20:12.405080 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.405031    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 18:20:12.405658 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.405649    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 18:20:12.409044 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.409033    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 18:20:12.409175 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.409166    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 18:20:12.409747 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.409739    2573 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 18:20:12.409774 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.409752    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 18:20:12.409774 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.409762    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 18:20:12.409774 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.409771    2573 kubelet.go:405] "Adding apiserver pod source"
Oct 31 18:20:12.409863 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.409778    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 18:20:12.410172 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.410160    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 18:20:12.411958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.411944    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 18:20:12.415454 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.415442    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 18:20:12.415986 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.415974    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 18:20:12.416019 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416005    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 18:20:12.416199 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416189    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 18:20:12.416227 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416205    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 18:20:12.416227 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416212    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 18:20:12.416227 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416217    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 18:20:12.416227 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416225    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 18:20:12.416322 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416235    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 18:20:12.416600 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416591    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 18:20:12.416627 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416601    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 18:20:12.416627 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416609    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 18:20:12.416627 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416614    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 18:20:12.416627 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416622    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 18:20:12.416746 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.416629    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 18:20:12.417041 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.417033    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 18:20:12.417090 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.417041    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 18:20:12.420512 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.420499    2573 server.go:1263] "Started kubelet"
Oct 31 18:20:12.420601 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.420579    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 18:20:12.422823 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.420723    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 18:20:12.422939 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.422914    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 18:20:12.422983 ip-10-0-142-120 systemd[1]: Started Kubernetes Kubelet.
Oct 31 18:20:12.424480 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.424464    2573 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 18:20:12.426289 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.426273    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 18:20:12.427998 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:12.427981    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 18:20:12.429139 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.429120    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 18:20:12.430102 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.430087    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 18:20:12.430247 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:12.430222    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-120.us-west-1.compute.internal\" not found"
Oct 31 18:20:12.430728 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.430712    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 18:20:12.430728 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.430713    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 18:20:12.430823 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.430739    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 18:20:12.430876 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.430824    2573 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 18:20:12.430876 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.430832    2573 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 18:20:12.431555 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.431540    2573 factory.go:55] Registering systemd factory
Oct 31 18:20:12.431614 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.431559    2573 factory.go:223] Registration of the systemd container factory successfully
Oct 31 18:20:12.431781 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.431769    2573 factory.go:153] Registering CRI-O factory
Oct 31 18:20:12.431816 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.431784    2573 factory.go:223] Registration of the crio container factory successfully
Oct 31 18:20:12.431842 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.431823    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 18:20:12.431842 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.431837    2573 factory.go:103] Registering Raw factory
Oct 31 18:20:12.431934 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.431848    2573 manager.go:1199] Started watching for new ooms in manager
Oct 31 18:20:12.432652 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.432645    2573 manager.go:319] Starting recovery of all containers
Oct 31 18:20:12.442251 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.442234    2573 manager.go:324] Recovery completed
Oct 31 18:20:12.443896 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:12.443875    2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 31 18:20:12.446653 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.446642    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:12.448109 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.448042    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:12.448191 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.448121    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:12.448191 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.448144    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:12.448661 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.448646    2573 cpu_manager.go:226] "Starting" policy="none"
Oct 31 18:20:12.448661 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.448658    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 18:20:12.448761 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.448676    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 18:20:12.448761 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.448698    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:12.449449 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.449435    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:12.449502 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.449459    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:12.449502 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.449472    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:12.450873 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.450859    2573 policy_none.go:50] "Start"
Oct 31 18:20:12.450942 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.450878    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 18:20:12.450942 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.450891    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 18:20:12.452945 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.452931    2573 policy_none.go:44] "Start"
Oct 31 18:20:12.490645 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.490613    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 18:20:12.493223 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.493088    2573 manager.go:341] "Starting Device Plugin manager"
Oct 31 18:20:12.493348 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:12.493237    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 18:20:12.493348 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.493246    2573 server.go:87] "Starting device plugin registration server"
Oct 31 18:20:12.493445 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.493428    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 18:20:12.493492 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.493437    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 18:20:12.493549 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.493531    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 18:20:12.493648 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.493633    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 18:20:12.493754 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.493744    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 18:20:12.494115 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:12.494096    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 18:20:12.494197 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:12.494147    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-120.us-west-1.compute.internal\" not found"
Oct 31 18:20:12.569886 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.569857    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 18:20:12.571819 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.571186    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 18:20:12.571819 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.571206    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 18:20:12.571819 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.571223    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 18:20:12.571819 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:12.571264    2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 18:20:12.594235 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.594216    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:12.594940 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.594923    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:12.595010 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.594955    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:12.595010 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.594968    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:12.595010 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.594992    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.672065 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.672032    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"]
Oct 31 18:20:12.672163 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.672128    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:12.673068 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.673028    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:12.673153 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.673082    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:12.673153 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.673098    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:12.675269 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.675257    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:12.675422 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.675405    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.675479 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.675446    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:12.676120 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.676104    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:12.676191 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.676136    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:12.676191 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.676148    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:12.676191 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.676104    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:12.676191 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.676184    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:12.676376 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.676200    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:12.678249 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.678233    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.678304 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.678266    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:12.679021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.679007    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:12.679111 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.679029    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:12.679111 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.679038    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:12.699874 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.699856    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 18:20:12.699956 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.699931    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t6nhc"
Oct 31 18:20:12.718475 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:12.718458    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-120.us-west-1.compute.internal\" not found" node="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.721161 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:12.721142    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-120.us-west-1.compute.internal\" not found" node="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.722153 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:12.722125    2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-120.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 18:20:12.722227 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.722184    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-120.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 18:20:12.722828 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:12.722126    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-120.us-west-1.compute.internal.18e3b033a15a84eb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-120.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-120.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 18:20:12.420474091 +0000 UTC m=+0.297014386,LastTimestamp:2026-10-31 18:20:12.420474091 +0000 UTC m=+0.297014386,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-120.us-west-1.compute.internal,}"
Oct 31 18:20:12.832010 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.831985    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fabc6b6751b80acc2b46148e9c9b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"26fabc6b6751b80acc2b46148e9c9b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.832010 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.832012    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb7aca0bcf277ba7408a15c0b72c2378-config\") pod \"kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"eb7aca0bcf277ba7408a15c0b72c2378\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.832181 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.832028    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26fabc6b6751b80acc2b46148e9c9b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"26fabc6b6751b80acc2b46148e9c9b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.876942 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.876896    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:12.877002 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.876976    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:12.878827 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.878806    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:12.932901 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.932879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26fabc6b6751b80acc2b46148e9c9b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"26fabc6b6751b80acc2b46148e9c9b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.932998 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.932910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fabc6b6751b80acc2b46148e9c9b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"26fabc6b6751b80acc2b46148e9c9b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.932998 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.932932    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb7aca0bcf277ba7408a15c0b72c2378-config\") pod \"kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"eb7aca0bcf277ba7408a15c0b72c2378\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.932998 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.932991    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fabc6b6751b80acc2b46148e9c9b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"26fabc6b6751b80acc2b46148e9c9b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.933141 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.932993    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb7aca0bcf277ba7408a15c0b72c2378-config\") pod \"kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"eb7aca0bcf277ba7408a15c0b72c2378\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.933141 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.932991    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26fabc6b6751b80acc2b46148e9c9b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"26fabc6b6751b80acc2b46148e9c9b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.975399 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:12.975375    2573 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-142-120.us-west-1.compute.internal\" not found" node="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:12.976364 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:12.976348    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:13.021598 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.021580    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:13.023740 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.023725    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:13.026356 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.026066    2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-120.us-west-1.compute.internal\" not found" node="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:13.176152 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.176088    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:13.177026 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.177009    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:13.177153 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.177044    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:13.177153 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.177075    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:13.177153 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.177115    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:13.232974 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.232947    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:13.330295 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.330271    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:13.372676 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.372653    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 18:20:13.373147 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.372758    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 18:20:13.373147 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.372792    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 18:20:13.373147 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.372841    2573 kubelet.go:3370] "Failed creating a mirror pod" err="Post \"https://a7971b018aa4c4793a97432a2228923c-30a6b095e3cfe1a3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.142.120:42036->3.133.177.27:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:13.373147 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.372869    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:13.373147 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.372844    2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a7971b018aa4c4793a97432a2228923c-30a6b095e3cfe1a3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-142-120.us-west-1.compute.internal.18e3b033a3002fba\": read tcp 10.0.142.120:42036->3.133.177.27:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-142-120.us-west-1.compute.internal.18e3b033a3002fba  default   4897 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-120.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-120.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-120.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 18:20:12 +0000 UTC,LastTimestamp:2026-10-31 18:20:12.594939801 +0000 UTC m=+0.471480095,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-120.us-west-1.compute.internal,}"
Oct 31 18:20:13.373147 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.372856    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 18:20:13.410403 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.410384    2573 apiserver.go:52] "Watching apiserver"
Oct 31 18:20:13.429813 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.429774    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 18:20:13.676336 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.676309    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 18:20:13.676851 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.676834    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 18:20:13.677182 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.677163    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-82t6b","openshift-dns/node-resolver-tjxmm","openshift-image-registry/node-ca-g8ng5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6","openshift-cluster-node-tuning-operator/tuned-gpjxn","openshift-network-diagnostics/network-check-target-9mlcf","openshift-ovn-kubernetes/ovnkube-node-vsrqn","kube-system/global-pull-secret-syncer-h72m5","openshift-multus/multus-nlk9v","openshift-multus/network-metrics-daemon-r92q2","openshift-network-operator/iptables-alerter-rjntd","openshift-multus/multus-additional-cni-plugins-xpvmh"]
Oct 31 18:20:13.679611 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.679597    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:13.681615 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.681578    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:13.683532 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.683515    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:13.685543 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.685526    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.687548 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.687535    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.689496 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.689483    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:13.689557 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.689542    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:13.691736 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.691723    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.693815 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.693801    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:13.693877 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.693845    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:13.695910 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.695898    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.698111 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.698097    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:13.698173 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.698141    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:13.700401 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.700386    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:13.702660 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.702645    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"]
Oct 31 18:20:13.702728 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.702718    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.704119 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.704084    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 18:15:14 +0000 UTC" deadline="2028-05-10 15:19:46.051233311 +0000 UTC"
Oct 31 18:20:13.704176 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.704119    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13364h59m32.347116578s"
Oct 31 18:20:13.731569 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.731553    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 18:20:13.731762 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.731749    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 18:20:13.732010 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.731998    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 18:20:13.732010 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.732005    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m5pkq\""
Oct 31 18:20:13.732834 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.732818    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 18:20:13.733403 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.733374    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 18:20:13.733489 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.733393    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xr757\""
Oct 31 18:20:13.736125 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736105    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-systemd-units\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.736193 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736129    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-ovn\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.736193 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736145    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-cni-bin\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.736193 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736159    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-cni-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.736193 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736180    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/494630d1-d540-42a4-93ab-2b76505246a5-cni-binary-copy\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.736305 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736222    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz9zx\" (UniqueName: \"kubernetes.io/projected/32e9756e-22b8-4adb-a36f-dc837581e01a-kube-api-access-vz9zx\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:13.736305 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736250    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-tuned\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.736305 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736266    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-netns\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.736305 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736289    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2zvs\" (UniqueName: \"kubernetes.io/projected/828b1043-03ed-45a2-afb3-160da3dce8c0-kube-api-access-g2zvs\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.736305 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736303    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-cni-bin\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.736443 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736353    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-conf-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.736443 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736398    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/207d8f7f-20e9-430d-a74d-42419bc4da86-iptables-alerter-script\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:13.736443 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736429    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-registration-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.736580 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736460    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-device-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.736580 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736500    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-cni-netd\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.736580 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736531    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-modprobe-d\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.736580 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736556    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-k8s-cni-cncf-io\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.736727 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736578    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-systemd\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.736727 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736600    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a79a203-4e31-41e5-af54-c8d07320badd-ovn-node-metrics-cert\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.736727 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736623    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-systemd\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.736727 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736649    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl47l\" (UniqueName: \"kubernetes.io/projected/494630d1-d540-42a4-93ab-2b76505246a5-kube-api-access-zl47l\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.736727 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736687    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hcq9\" (UniqueName: \"kubernetes.io/projected/207d8f7f-20e9-430d-a74d-42419bc4da86-kube-api-access-5hcq9\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:13.736727 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736705    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cni-binary-copy\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.736930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736749    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhf2p\" (UniqueName: \"kubernetes.io/projected/203b8b56-b634-4a8b-85b1-46aae63f6dd8-kube-api-access-rhf2p\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.736930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0b37043-a08e-475d-b8ff-3c8c2c8b653f-agent-certs\") pod \"konnectivity-agent-82t6b\" (UID: \"e0b37043-a08e-475d-b8ff-3c8c2c8b653f\") " pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:13.736930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736794    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e9756e-22b8-4adb-a36f-dc837581e01a-tmp-dir\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:13.736930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736821    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bfedf5f-370f-425e-88f8-0959db69e660-serviceca\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:13.736930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736835    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-kubelet\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.736930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736858    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.736930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736873    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-kubelet-config\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:13.736930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-etc-kubernetes\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.737233 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736942    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7jq\" (UniqueName: \"kubernetes.io/projected/3bfedf5f-370f-425e-88f8-0959db69e660-kube-api-access-xn7jq\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:13.737233 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.736980    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.737233 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737022    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-run\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.737233 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737071    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-run-ovn-kubernetes\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.737233 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737098    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysconfig\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.737233 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737125    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-system-cni-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.737233 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737158    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0b37043-a08e-475d-b8ff-3c8c2c8b653f-konnectivity-ca\") pod \"konnectivity-agent-82t6b\" (UID: \"e0b37043-a08e-475d-b8ff-3c8c2c8b653f\") " pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:13.737233 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.737233 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737198    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-slash\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.737233 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737235    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-dbus\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737256    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-hostroot\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737271    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32e9756e-22b8-4adb-a36f-dc837581e01a-hosts-file\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737298    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-sys-fs\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737313    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-log-socket\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737338    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-env-overrides\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737353    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-ovnkube-script-lib\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737370    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92g26\" (UniqueName: \"kubernetes.io/projected/ac38de34-5c48-430d-b06f-7015c64fbb56-kube-api-access-92g26\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737386    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-lib-modules\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cnibin\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737450    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-socket-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737485    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737504    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-kubernetes\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysctl-d\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737592    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/207d8f7f-20e9-430d-a74d-42419bc4da86-host-slash\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:13.737623 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737619    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737643    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737667    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-cnibin\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737689    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/494630d1-d540-42a4-93ab-2b76505246a5-multus-daemon-config\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737702    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737720    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737735    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-host\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737749    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r68bd\" (UniqueName: \"kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd\") pod \"network-check-target-9mlcf\" (UID: \"b4b27f4e-abf1-407d-a33d-90e23a5fc861\") " pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737778    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-kubelet\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737794    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-system-cni-dir\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737835    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpzsx\" (UniqueName: \"kubernetes.io/projected/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-kube-api-access-zpzsx\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737865    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-node-log\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737881    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg9rk\" (UniqueName: \"kubernetes.io/projected/0a79a203-4e31-41e5-af54-c8d07320badd-kube-api-access-gg9rk\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737899    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737913    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bfedf5f-370f-425e-88f8-0959db69e660-host\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737949    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-var-lib-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.738160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.737978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysctl-conf\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.738742 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.738000    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-sys\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.738742 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.738038    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-socket-dir-parent\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.738742 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.738081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-multus-certs\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.738742 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.738100    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-run-netns\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.738742 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.738125    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-etc-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.738742 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.738155    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-ovnkube-config\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.738742 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.738182    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-var-lib-kubelet\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.738742 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.738197    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/828b1043-03ed-45a2-afb3-160da3dce8c0-tmp\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.738742 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.738219    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-cni-multus\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.828936 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.828915    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 18:20:13.829083 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.828982    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-khz4z"
Oct 31 18:20:13.832454 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.832435    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 18:20:13.832542 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.832479    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 18:20:13.832542 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.832486    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 18:20:13.832742 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.832727    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 18:20:13.832810 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.832793    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 18:20:13.832853 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.832830    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 18:20:13.832898 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.832865    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 18:20:13.832939 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.832897    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 18:20:13.832974 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.832953    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 18:20:13.833039 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.832987    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 18:20:13.833039 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.833005    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 18:20:13.833339 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.833325    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 18:20:13.833411 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.833327    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 18:20:13.833411 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.833377    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 18:20:13.833757 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.833657    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 18:20:13.833757 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.833660    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 18:20:13.833757 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.833700    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-72cf5\""
Oct 31 18:20:13.833757 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.833700    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 18:20:13.833757 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.833731    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgxxf\""
Oct 31 18:20:13.833757 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.833732    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w82k4\""
Oct 31 18:20:13.833757 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.833751    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 18:20:13.834143 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.833781    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9xfhz\""
Oct 31 18:20:13.839255 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839238    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-var-lib-kubelet\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.839336 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839262    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/828b1043-03ed-45a2-afb3-160da3dce8c0-tmp\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.839336 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839279    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-cni-multus\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.839336 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839296    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-systemd-units\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.839336 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839317    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-ovn\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.839500 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-cni-bin\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.839500 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839349    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-var-lib-kubelet\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.839500 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839380    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-cni-multus\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.839500 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839393    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-cni-bin\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.839500 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839385    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-ovn\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.839500 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839409    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-systemd-units\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.839500 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-cni-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.839500 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/494630d1-d540-42a4-93ab-2b76505246a5-cni-binary-copy\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.839500 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz9zx\" (UniqueName: \"kubernetes.io/projected/32e9756e-22b8-4adb-a36f-dc837581e01a-kube-api-access-vz9zx\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839520    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-tuned\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839544    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-netns\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839545    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-cni-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839568    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839600    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-netns\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zvs\" (UniqueName: \"kubernetes.io/projected/828b1043-03ed-45a2-afb3-160da3dce8c0-kube-api-access-g2zvs\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839649    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-cni-bin\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-conf-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839691    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/207d8f7f-20e9-430d-a74d-42419bc4da86-iptables-alerter-script\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-registration-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839738    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-device-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839761    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-cni-netd\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.839778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839783    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-modprobe-d\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839775    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-conf-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839805    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-k8s-cni-cncf-io\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839827    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-systemd\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839842    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-cni-bin\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a79a203-4e31-41e5-af54-c8d07320badd-ovn-node-metrics-cert\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839873    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-systemd\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839909    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-cni-netd\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839927    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-systemd\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839937    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl47l\" (UniqueName: \"kubernetes.io/projected/494630d1-d540-42a4-93ab-2b76505246a5-kube-api-access-zl47l\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.839984    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcq9\" (UniqueName: \"kubernetes.io/projected/207d8f7f-20e9-430d-a74d-42419bc4da86-kube-api-access-5hcq9\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840006    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-systemd\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840012    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cni-binary-copy\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840030    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/494630d1-d540-42a4-93ab-2b76505246a5-cni-binary-copy\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840046    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-k8s-cni-cncf-io\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840066    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhf2p\" (UniqueName: \"kubernetes.io/projected/203b8b56-b634-4a8b-85b1-46aae63f6dd8-kube-api-access-rhf2p\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840097    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0b37043-a08e-475d-b8ff-3c8c2c8b653f-agent-certs\") pod \"konnectivity-agent-82t6b\" (UID: \"e0b37043-a08e-475d-b8ff-3c8c2c8b653f\") " pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840118    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e9756e-22b8-4adb-a36f-dc837581e01a-tmp-dir\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:13.840451 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bfedf5f-370f-425e-88f8-0959db69e660-serviceca\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840157    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-kubelet\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840182    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840207    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-kubelet-config\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-etc-kubernetes\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840027    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-modprobe-d\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840254    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7jq\" (UniqueName: \"kubernetes.io/projected/3bfedf5f-370f-425e-88f8-0959db69e660-kube-api-access-xn7jq\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840275    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840296    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-run\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840314    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-run-ovn-kubernetes\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840321    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-registration-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysconfig\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840352    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-system-cni-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840372    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0b37043-a08e-475d-b8ff-3c8c2c8b653f-konnectivity-ca\") pod \"konnectivity-agent-82t6b\" (UID: \"e0b37043-a08e-475d-b8ff-3c8c2c8b653f\") " pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840390    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840397    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-kubelet-config\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-slash\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.841320 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840432    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-dbus\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840467    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-hostroot\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840486    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32e9756e-22b8-4adb-a36f-dc837581e01a-hosts-file\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840497    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-kubelet\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840505    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-sys-fs\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840525    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-log-socket\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840537    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840544    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-env-overrides\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840563    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-ovnkube-script-lib\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840584    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92g26\" (UniqueName: \"kubernetes.io/projected/ac38de34-5c48-430d-b06f-7015c64fbb56-kube-api-access-92g26\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840623    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-lib-modules\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840697    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cni-binary-copy\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840882    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cnibin\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840918    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-etc-kubernetes\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-socket-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840982    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-device-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.840986    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:13.842093 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.841124    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.841281    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs podName:ac38de34-5c48-430d-b06f-7015c64fbb56 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:14.341241233 +0000 UTC m=+2.217781533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs") pod "network-metrics-daemon-r92q2" (UID: "ac38de34-5c48-430d-b06f-7015c64fbb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841518    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e9756e-22b8-4adb-a36f-dc837581e01a-tmp-dir\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841031    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-kubernetes\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysctl-d\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841598    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/207d8f7f-20e9-430d-a74d-42419bc4da86-host-slash\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841649    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-run-ovn-kubernetes\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841684    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-cnibin\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841698    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysconfig\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841707    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/494630d1-d540-42a4-93ab-2b76505246a5-multus-daemon-config\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841745    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-system-cni-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.843021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841786    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cnibin\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841795    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-host\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841824    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68bd\" (UniqueName: \"kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd\") pod \"network-check-target-9mlcf\" (UID: \"b4b27f4e-abf1-407d-a33d-90e23a5fc861\") " pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-kubelet\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841875    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-system-cni-dir\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpzsx\" (UniqueName: \"kubernetes.io/projected/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-kube-api-access-zpzsx\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841935    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-node-log\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841939    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0b37043-a08e-475d-b8ff-3c8c2c8b653f-konnectivity-ca\") pod \"konnectivity-agent-82t6b\" (UID: \"e0b37043-a08e-475d-b8ff-3c8c2c8b653f\") " pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg9rk\" (UniqueName: \"kubernetes.io/projected/0a79a203-4e31-41e5-af54-c8d07320badd-kube-api-access-gg9rk\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841987    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.841992    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-env-overrides\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842010    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bfedf5f-370f-425e-88f8-0959db69e660-host\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842034    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-var-lib-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842074    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysctl-conf\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842043    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-socket-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842100    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-sys\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842104    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.843794 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-socket-dir-parent\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842139    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-lib-modules\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842154    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-kubernetes\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842157    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-sys\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842194    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-hostroot\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842219    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-sys-fs\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842248    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-multus-certs\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842269    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysctl-d\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842409    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-host\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842430    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-log-socket\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842517    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-cnibin\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842561    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842603    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysctl-conf\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842642    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-socket-dir-parent\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842678    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-slash\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842726    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-run-netns\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842784    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-etc-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842815    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-ovnkube-config\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.844582 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842887    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.842986    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-dbus\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843025    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-multus-certs\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843162    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/494630d1-d540-42a4-93ab-2b76505246a5-multus-daemon-config\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843220    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-var-lib-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843255    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-system-cni-dir\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843270    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843280    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-run\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843306    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bfedf5f-370f-425e-88f8-0959db69e660-host\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843312    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/207d8f7f-20e9-430d-a74d-42419bc4da86-host-slash\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843447    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-ovnkube-config\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843576    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/828b1043-03ed-45a2-afb3-160da3dce8c0-tmp\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843716    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32e9756e-22b8-4adb-a36f-dc837581e01a-hosts-file\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843775    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-node-log\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843819    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a79a203-4e31-41e5-af54-c8d07320badd-ovn-node-metrics-cert\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843851    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-run-netns\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843821    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-etc-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843874    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.845084 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843909    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-kubelet\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.845638 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.843933    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-tuned\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.845638 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.843968    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:13.845638 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.844020    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret podName:c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:14.34400213 +0000 UTC m=+2.220542426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret") pod "global-pull-secret-syncer-h72m5" (UID: "c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:13.845638 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.844158    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-szhfk\""
Oct 31 18:20:13.845638 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.844179    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 18:20:13.845638 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.844164    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 18:20:13.845638 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.844218    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 18:20:13.845638 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.844291    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 18:20:13.845638 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.844322    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-27pxv\""
Oct 31 18:20:13.845638 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.844423    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l4kdr\""
Oct 31 18:20:13.845638 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.844593    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 18:20:13.845638 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.845085    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0b37043-a08e-475d-b8ff-3c8c2c8b653f-agent-certs\") pod \"konnectivity-agent-82t6b\" (UID: \"e0b37043-a08e-475d-b8ff-3c8c2c8b653f\") " pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:13.851168 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.851149    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/207d8f7f-20e9-430d-a74d-42419bc4da86-iptables-alerter-script\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:13.851245 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.851192    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bfedf5f-370f-425e-88f8-0959db69e660-serviceca\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:13.852996 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.852979    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-ovnkube-script-lib\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.896347 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.896318    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz9zx\" (UniqueName: \"kubernetes.io/projected/32e9756e-22b8-4adb-a36f-dc837581e01a-kube-api-access-vz9zx\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:13.896630 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.896614    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:13.896678 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.896633    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:13.896678 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.896644    2573 projected.go:196] Error preparing data for projected volume kube-api-access-r68bd for pod openshift-network-diagnostics/network-check-target-9mlcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:13.896765 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:13.896730    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd podName:b4b27f4e-abf1-407d-a33d-90e23a5fc861 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:14.396713106 +0000 UTC m=+2.273253390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r68bd" (UniqueName: "kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd") pod "network-check-target-9mlcf" (UID: "b4b27f4e-abf1-407d-a33d-90e23a5fc861") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:13.896993 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.896977    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhf2p\" (UniqueName: \"kubernetes.io/projected/203b8b56-b634-4a8b-85b1-46aae63f6dd8-kube-api-access-rhf2p\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:13.897043 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.896979    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zvs\" (UniqueName: \"kubernetes.io/projected/828b1043-03ed-45a2-afb3-160da3dce8c0-kube-api-access-g2zvs\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:13.897218 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.897198    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl47l\" (UniqueName: \"kubernetes.io/projected/494630d1-d540-42a4-93ab-2b76505246a5-kube-api-access-zl47l\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:13.897844 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.897819    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92g26\" (UniqueName: \"kubernetes.io/projected/ac38de34-5c48-430d-b06f-7015c64fbb56-kube-api-access-92g26\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:13.899264 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.899240    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg9rk\" (UniqueName: \"kubernetes.io/projected/0a79a203-4e31-41e5-af54-c8d07320badd-kube-api-access-gg9rk\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:13.899547 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.899525    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7jq\" (UniqueName: \"kubernetes.io/projected/3bfedf5f-370f-425e-88f8-0959db69e660-kube-api-access-xn7jq\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:13.900727 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.900705    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hcq9\" (UniqueName: \"kubernetes.io/projected/207d8f7f-20e9-430d-a74d-42419bc4da86-kube-api-access-5hcq9\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:13.900930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:13.900905    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpzsx\" (UniqueName: \"kubernetes.io/projected/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-kube-api-access-zpzsx\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:14.017014 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.016936    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:14.022159 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.022136    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:14.045774 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.045744    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:14.062516 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.062495    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:14.072081 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.072065    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:14.096643 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.096626    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:14.102807 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.102793    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:14.118458 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.118439    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:14.129892 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:14.129860    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494630d1_d540_42a4_93ab_2b76505246a5.slice/crio-4db540bbaaa0bc0204d97f5a4aa44acfd0c096e4b2ab77c7acd0db0532f1c614 WatchSource:0}: Error finding container 4db540bbaaa0bc0204d97f5a4aa44acfd0c096e4b2ab77c7acd0db0532f1c614: Status 404 returned error can't find the container with id 4db540bbaaa0bc0204d97f5a4aa44acfd0c096e4b2ab77c7acd0db0532f1c614
Oct 31 18:20:14.130267 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:14.130247    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a79a203_4e31_41e5_af54_c8d07320badd.slice/crio-a330e580cefaa0c74e25f929a39ff7e9c7dfddebfc34f81473e0eea460c3b722 WatchSource:0}: Error finding container a330e580cefaa0c74e25f929a39ff7e9c7dfddebfc34f81473e0eea460c3b722: Status 404 returned error can't find the container with id a330e580cefaa0c74e25f929a39ff7e9c7dfddebfc34f81473e0eea460c3b722
Oct 31 18:20:14.131499 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:14.131473    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fabc6b6751b80acc2b46148e9c9b23.slice/crio-88be93709b51a75f67c205065e5a283c303d550473e9cc04242460d77968583b WatchSource:0}: Error finding container 88be93709b51a75f67c205065e5a283c303d550473e9cc04242460d77968583b: Status 404 returned error can't find the container with id 88be93709b51a75f67c205065e5a283c303d550473e9cc04242460d77968583b
Oct 31 18:20:14.133270 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:14.133242    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7aca0bcf277ba7408a15c0b72c2378.slice/crio-d9817bf28fe482befd8dafed99e9f4250e49297303808ffa69020c612cae69e9 WatchSource:0}: Error finding container d9817bf28fe482befd8dafed99e9f4250e49297303808ffa69020c612cae69e9: Status 404 returned error can't find the container with id d9817bf28fe482befd8dafed99e9f4250e49297303808ffa69020c612cae69e9
Oct 31 18:20:14.134208 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.134188    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:14.134372 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:14.134351    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207d8f7f_20e9_430d_a74d_42419bc4da86.slice/crio-1fbe0e2b0a44ecb26d14f16c42bccce270e1c85bf6f0499c926351a78c2dcaaa WatchSource:0}: Error finding container 1fbe0e2b0a44ecb26d14f16c42bccce270e1c85bf6f0499c926351a78c2dcaaa: Status 404 returned error can't find the container with id 1fbe0e2b0a44ecb26d14f16c42bccce270e1c85bf6f0499c926351a78c2dcaaa
Oct 31 18:20:14.136777 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.136768    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:20:14.143187 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:14.143169    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203b8b56_b634_4a8b_85b1_46aae63f6dd8.slice/crio-26745ba601d694844e1c08573e13763f0f2cdcc9e3fa1b5112b8ea65facb680d WatchSource:0}: Error finding container 26745ba601d694844e1c08573e13763f0f2cdcc9e3fa1b5112b8ea65facb680d: Status 404 returned error can't find the container with id 26745ba601d694844e1c08573e13763f0f2cdcc9e3fa1b5112b8ea65facb680d
Oct 31 18:20:14.251522 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.251491    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:14.345598 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.345532    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:14.345598 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.345576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:14.345786 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:14.345661    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:14.345786 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:14.345683    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:14.345786 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:14.345706    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs podName:ac38de34-5c48-430d-b06f-7015c64fbb56 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:15.345692919 +0000 UTC m=+3.222233200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs") pod "network-metrics-daemon-r92q2" (UID: "ac38de34-5c48-430d-b06f-7015c64fbb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:14.345786 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:14.345735    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret podName:c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:15.345718925 +0000 UTC m=+3.222259214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret") pod "global-pull-secret-syncer-h72m5" (UID: "c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:14.446117 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.446091    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68bd\" (UniqueName: \"kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd\") pod \"network-check-target-9mlcf\" (UID: \"b4b27f4e-abf1-407d-a33d-90e23a5fc861\") " pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:14.446650 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:14.446197    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:14.446650 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:14.446209    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:14.446650 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:14.446217    2573 projected.go:196] Error preparing data for projected volume kube-api-access-r68bd for pod openshift-network-diagnostics/network-check-target-9mlcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:14.446650 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:14.446257    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd podName:b4b27f4e-abf1-407d-a33d-90e23a5fc861 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:15.446245331 +0000 UTC m=+3.322785625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r68bd" (UniqueName: "kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd") pod "network-check-target-9mlcf" (UID: "b4b27f4e-abf1-407d-a33d-90e23a5fc861") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:14.521173 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:14.521144    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:16.074005 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:16.073955    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nlk9v" event={"ID":"494630d1-d540-42a4-93ab-2b76505246a5","Type":"ContainerStarted","Data":"4db540bbaaa0bc0204d97f5a4aa44acfd0c096e4b2ab77c7acd0db0532f1c614"}
Oct 31 18:20:16.075009 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:16.074990    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn" event={"ID":"0a79a203-4e31-41e5-af54-c8d07320badd","Type":"ContainerStarted","Data":"a330e580cefaa0c74e25f929a39ff7e9c7dfddebfc34f81473e0eea460c3b722"}
Oct 31 18:20:16.075947 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:16.075927    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpvmh" event={"ID":"203b8b56-b634-4a8b-85b1-46aae63f6dd8","Type":"ContainerStarted","Data":"26745ba601d694844e1c08573e13763f0f2cdcc9e3fa1b5112b8ea65facb680d"}
Oct 31 18:20:16.076807 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:16.076791    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rjntd" event={"ID":"207d8f7f-20e9-430d-a74d-42419bc4da86","Type":"ContainerStarted","Data":"1fbe0e2b0a44ecb26d14f16c42bccce270e1c85bf6f0499c926351a78c2dcaaa"}
Oct 31 18:20:16.077703 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:16.077684    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal" event={"ID":"eb7aca0bcf277ba7408a15c0b72c2378","Type":"ContainerStarted","Data":"d9817bf28fe482befd8dafed99e9f4250e49297303808ffa69020c612cae69e9"}
Oct 31 18:20:16.078537 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:16.078519    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal" event={"ID":"26fabc6b6751b80acc2b46148e9c9b23","Type":"ContainerStarted","Data":"88be93709b51a75f67c205065e5a283c303d550473e9cc04242460d77968583b"}
Oct 31 18:20:16.175295 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:16.175228    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:16.320258 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:16.320232    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b37043_a08e_475d_b8ff_3c8c2c8b653f.slice/crio-dac23bbe9845c5aa0c5a9293b3e821f6f61a65331aab689755505b3f439a455b WatchSource:0}: Error finding container dac23bbe9845c5aa0c5a9293b3e821f6f61a65331aab689755505b3f439a455b: Status 404 returned error can't find the container with id dac23bbe9845c5aa0c5a9293b3e821f6f61a65331aab689755505b3f439a455b
Oct 31 18:20:16.330021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:16.329985    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 18:15:15 +0000 UTC" deadline="2028-04-21 08:33:07.513588281 +0000 UTC"
Oct 31 18:20:16.330021 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:16.330021    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12902h12m51.183569813s"
Oct 31 18:20:16.394394 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:16.394354    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e9756e_22b8_4adb_a36f_dc837581e01a.slice/crio-9d23880c4071a4c19ad010e4d9a9ca1335d7b6e4fe11d4df5dec397b2c02cafe WatchSource:0}: Error finding container 9d23880c4071a4c19ad010e4d9a9ca1335d7b6e4fe11d4df5dec397b2c02cafe: Status 404 returned error can't find the container with id 9d23880c4071a4c19ad010e4d9a9ca1335d7b6e4fe11d4df5dec397b2c02cafe
Oct 31 18:20:16.430516 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:16.430490    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfedf5f_370f_425e_88f8_0959db69e660.slice/crio-5c21fc7a7689238d0ddf56d693b8c24f2dfb3b5fc4d85c3c522a28944e2575cf WatchSource:0}: Error finding container 5c21fc7a7689238d0ddf56d693b8c24f2dfb3b5fc4d85c3c522a28944e2575cf: Status 404 returned error can't find the container with id 5c21fc7a7689238d0ddf56d693b8c24f2dfb3b5fc4d85c3c522a28944e2575cf
Oct 31 18:20:16.431237 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:16.431218    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828b1043_03ed_45a2_afb3_160da3dce8c0.slice/crio-58dff7afa74bcc0caf1a9e88b4dae784efae76978be993c8421a63b2a34b41e1 WatchSource:0}: Error finding container 58dff7afa74bcc0caf1a9e88b4dae784efae76978be993c8421a63b2a34b41e1: Status 404 returned error can't find the container with id 58dff7afa74bcc0caf1a9e88b4dae784efae76978be993c8421a63b2a34b41e1
Oct 31 18:20:16.478438 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:16.478414    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c12bcbd_bf78_49c1_b60c_0c39e16faf61.slice/crio-0acb523aa6e0990bcdf23522a8e707265a3204c7cb9c4bad1c4a53145a0ee760 WatchSource:0}: Error finding container 0acb523aa6e0990bcdf23522a8e707265a3204c7cb9c4bad1c4a53145a0ee760: Status 404 returned error can't find the container with id 0acb523aa6e0990bcdf23522a8e707265a3204c7cb9c4bad1c4a53145a0ee760
Oct 31 18:20:16.851727 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:16.851367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:16.851727 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:16.851430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:16.851727 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:16.851551    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:16.851727 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:16.851597    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret podName:c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:18.851585495 +0000 UTC m=+5.229703782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret") pod "global-pull-secret-syncer-h72m5" (UID: "c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:16.851727 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:16.851643    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:16.851727 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:16.851663    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs podName:ac38de34-5c48-430d-b06f-7015c64fbb56 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:18.851656587 +0000 UTC m=+5.229774875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs") pod "network-metrics-daemon-r92q2" (UID: "ac38de34-5c48-430d-b06f-7015c64fbb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:16.952880 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:16.952609    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68bd\" (UniqueName: \"kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd\") pod \"network-check-target-9mlcf\" (UID: \"b4b27f4e-abf1-407d-a33d-90e23a5fc861\") " pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:16.952880 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:16.952757    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:16.952880 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:16.952773    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:16.952880 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:16.952785    2573 projected.go:196] Error preparing data for projected volume kube-api-access-r68bd for pod openshift-network-diagnostics/network-check-target-9mlcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:16.952880 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:16.952835    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd podName:b4b27f4e-abf1-407d-a33d-90e23a5fc861 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:18.952819518 +0000 UTC m=+5.330937807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r68bd" (UniqueName: "kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd") pod "network-check-target-9mlcf" (UID: "b4b27f4e-abf1-407d-a33d-90e23a5fc861") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:17.070567 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:17.070376    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:17.070567 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:17.070519    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:17.070811 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:17.070615    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:17.070811 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:17.070704    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:17.070811 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:17.070761    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:17.070952 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:17.070826    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:17.082713 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:17.082630    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6" event={"ID":"8c12bcbd-bf78-49c1-b60c-0c39e16faf61","Type":"ContainerStarted","Data":"0acb523aa6e0990bcdf23522a8e707265a3204c7cb9c4bad1c4a53145a0ee760"}
Oct 31 18:20:17.086929 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:17.086900    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gpjxn" event={"ID":"828b1043-03ed-45a2-afb3-160da3dce8c0","Type":"ContainerStarted","Data":"58dff7afa74bcc0caf1a9e88b4dae784efae76978be993c8421a63b2a34b41e1"}
Oct 31 18:20:17.096216 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:17.096149    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tjxmm" event={"ID":"32e9756e-22b8-4adb-a36f-dc837581e01a","Type":"ContainerStarted","Data":"9d23880c4071a4c19ad010e4d9a9ca1335d7b6e4fe11d4df5dec397b2c02cafe"}
Oct 31 18:20:17.100175 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:17.100144    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g8ng5" event={"ID":"3bfedf5f-370f-425e-88f8-0959db69e660","Type":"ContainerStarted","Data":"5c21fc7a7689238d0ddf56d693b8c24f2dfb3b5fc4d85c3c522a28944e2575cf"}
Oct 31 18:20:17.102736 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:17.102648    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-82t6b" event={"ID":"e0b37043-a08e-475d-b8ff-3c8c2c8b653f","Type":"ContainerStarted","Data":"dac23bbe9845c5aa0c5a9293b3e821f6f61a65331aab689755505b3f439a455b"}
Oct 31 18:20:17.330980 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:17.330918    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 18:15:15 +0000 UTC" deadline="2028-08-17 17:32:26.978493728 +0000 UTC"
Oct 31 18:20:17.330980 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:17.330977    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15743h12m9.647520188s"
Oct 31 18:20:18.868782 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:18.868718    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:18.868782 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:18.868763    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:18.869269 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:18.868876    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:18.869269 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:18.868916    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret podName:c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:22.868905553 +0000 UTC m=+9.247023829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret") pod "global-pull-secret-syncer-h72m5" (UID: "c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:18.869269 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:18.869175    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:18.869269 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:18.869204    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs podName:ac38de34-5c48-430d-b06f-7015c64fbb56 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:22.869195385 +0000 UTC m=+9.247313673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs") pod "network-metrics-daemon-r92q2" (UID: "ac38de34-5c48-430d-b06f-7015c64fbb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:18.969502 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:18.969202    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68bd\" (UniqueName: \"kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd\") pod \"network-check-target-9mlcf\" (UID: \"b4b27f4e-abf1-407d-a33d-90e23a5fc861\") " pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:18.969502 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:18.969379    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:18.969502 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:18.969400    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:18.969502 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:18.969409    2573 projected.go:196] Error preparing data for projected volume kube-api-access-r68bd for pod openshift-network-diagnostics/network-check-target-9mlcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:18.969502 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:18.969449    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd podName:b4b27f4e-abf1-407d-a33d-90e23a5fc861 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:22.969436676 +0000 UTC m=+9.347554964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r68bd" (UniqueName: "kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd") pod "network-check-target-9mlcf" (UID: "b4b27f4e-abf1-407d-a33d-90e23a5fc861") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:19.070375 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:19.070348    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:19.070556 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:19.070456    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:19.070790 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:19.070771    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:19.070875 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:19.070857    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:19.070930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:19.070904    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:19.070986 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:19.070965    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:20.129664 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:20.128418    2573 generic.go:356] "Generic (PLEG): container finished" podID="26fabc6b6751b80acc2b46148e9c9b23" containerID="f9a639036e9a74c230169c2ab53bf7fd10574f0a67faf4e736e343bf8388da23" exitCode=0
Oct 31 18:20:20.129664 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:20.129465    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal" event={"ID":"26fabc6b6751b80acc2b46148e9c9b23","Type":"ContainerDied","Data":"f9a639036e9a74c230169c2ab53bf7fd10574f0a67faf4e736e343bf8388da23"}
Oct 31 18:20:20.137742 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:20.137714    2573 generic.go:356] "Generic (PLEG): container finished" podID="203b8b56-b634-4a8b-85b1-46aae63f6dd8" containerID="6d977e44663439ed83b97b0307158a35d2a1918cab7f323dd64b876d551e31a6" exitCode=0
Oct 31 18:20:20.137877 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:20.137761    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpvmh" event={"ID":"203b8b56-b634-4a8b-85b1-46aae63f6dd8","Type":"ContainerDied","Data":"6d977e44663439ed83b97b0307158a35d2a1918cab7f323dd64b876d551e31a6"}
Oct 31 18:20:21.070347 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:21.070279    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:21.070527 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:21.070433    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:21.070586 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:21.070533    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:21.070631 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:21.070614    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:21.070962 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:21.070949    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:21.071047 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:21.071010    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:21.144919 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:21.144200    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal" event={"ID":"eb7aca0bcf277ba7408a15c0b72c2378","Type":"ContainerStarted","Data":"5bf28ecd9bca61815d391c14f9bcfc3bb453f30431715b85d9d0c87987503fd9"}
Oct 31 18:20:21.148288 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:21.148256    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal" event={"ID":"26fabc6b6751b80acc2b46148e9c9b23","Type":"ContainerStarted","Data":"af952c6d5252ab870afd96a35b3a3cc910db8c5b6618dd1bb387b451b7b48e33"}
Oct 31 18:20:21.151750 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:21.151689    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-82t6b" event={"ID":"e0b37043-a08e-475d-b8ff-3c8c2c8b653f","Type":"ContainerStarted","Data":"74c11620e44148eff8e003203acb4f5835ed16032aff5b170e33a0ea95b51d4c"}
Oct 31 18:20:21.259570 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:21.259507    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal" podStartSLOduration=6.259490361 podStartE2EDuration="6.259490361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:21.259145038 +0000 UTC m=+7.637263337" watchObservedRunningTime="2026-10-31 18:20:21.259490361 +0000 UTC m=+7.637608654"
Oct 31 18:20:21.492892 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:21.492785    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal" podStartSLOduration=7.492766312 podStartE2EDuration="7.492766312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:21.372405193 +0000 UTC m=+7.750523493" watchObservedRunningTime="2026-10-31 18:20:21.492766312 +0000 UTC m=+7.870884609"
Oct 31 18:20:22.158022 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:22.157996    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g8ng5" event={"ID":"3bfedf5f-370f-425e-88f8-0959db69e660","Type":"ContainerStarted","Data":"21db69fde49c288e7766d08647362e6e2fe7b897c75082e2480ae143c6878e5c"}
Oct 31 18:20:22.741038 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:22.741004    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:22.742404 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:22.742147    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:22.855714 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:22.855625    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-82t6b" podStartSLOduration=4.409518612 podStartE2EDuration="8.855606585s" totalImagesPullingTime="4.446087973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:21.493634535 +0000 UTC m=+7.871752835" watchObservedRunningTime="2026-10-31 18:20:22.855606585 +0000 UTC m=+9.233724884"
Oct 31 18:20:22.900492 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:22.900451    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:22.900644 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:22.900518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:22.900644 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:22.900622    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:22.900758 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:22.900666    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret podName:c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:30.90065473 +0000 UTC m=+17.278773005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret") pod "global-pull-secret-syncer-h72m5" (UID: "c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:22.900758 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:22.900697    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:22.900844 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:22.900763    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs podName:ac38de34-5c48-430d-b06f-7015c64fbb56 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:30.900744235 +0000 UTC m=+17.278862528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs") pod "network-metrics-daemon-r92q2" (UID: "ac38de34-5c48-430d-b06f-7015c64fbb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:23.001911 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:23.001836    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68bd\" (UniqueName: \"kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd\") pod \"network-check-target-9mlcf\" (UID: \"b4b27f4e-abf1-407d-a33d-90e23a5fc861\") " pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:23.002075 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:23.002052    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:23.002149 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:23.002078    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:23.002149 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:23.002092    2573 projected.go:196] Error preparing data for projected volume kube-api-access-r68bd for pod openshift-network-diagnostics/network-check-target-9mlcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:23.002149 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:23.002149    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd podName:b4b27f4e-abf1-407d-a33d-90e23a5fc861 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:31.002131643 +0000 UTC m=+17.380249932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r68bd" (UniqueName: "kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd") pod "network-check-target-9mlcf" (UID: "b4b27f4e-abf1-407d-a33d-90e23a5fc861") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:23.070245 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:23.070214    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:23.070411 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:23.070343    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:23.070464 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:23.070426    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:23.070579 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:23.070519    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:23.070579 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:23.070177    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:23.070664 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:23.070610    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:23.167754 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:23.167717    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6" event={"ID":"8c12bcbd-bf78-49c1-b60c-0c39e16faf61","Type":"ContainerStarted","Data":"6471ac7ea5f3ce77997dc17920bf10c2a3619cbcc6189bc391ee6e0a439cf822"}
Oct 31 18:20:24.171931 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:24.171901    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 18:20:24.172957 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:24.172894    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nlk9v" event={"ID":"494630d1-d540-42a4-93ab-2b76505246a5","Type":"ContainerStarted","Data":"8b35e5d9e858ea1c86edfd6c4ee1509196ff9dd4af0db82821c7560c66b15dbf"}
Oct 31 18:20:24.287119 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:24.287058    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g8ng5" podStartSLOduration=4.730788894 podStartE2EDuration="10.287037043s" totalImagesPullingTime="5.556248149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:23.280684089 +0000 UTC m=+9.658802387" watchObservedRunningTime="2026-10-31 18:20:24.287037043 +0000 UTC m=+10.665155340"
Oct 31 18:20:24.287735 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:24.287698    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nlk9v" podStartSLOduration=2.12505965 podStartE2EDuration="10.287687967s" totalImagesPullingTime="8.162628317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:24.286910059 +0000 UTC m=+10.665028357" watchObservedRunningTime="2026-10-31 18:20:24.287687967 +0000 UTC m=+10.665806266"
Oct 31 18:20:24.691834 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:24.691811    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 18:20:24.997258 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:24.997153    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T18:20:24.691831824Z","UUID":"58b4a1f1-f3ab-47f1-9a17-c1b47f13dac2","Handler":null,"Name":"","Endpoint":""}
Oct 31 18:20:24.998896 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:24.998720    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 18:20:24.999027 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:24.998904    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 18:20:25.070756 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:25.070687    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:25.070756 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:25.070708    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:25.070958 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:25.070804    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:25.070958 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:25.070884    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:25.070958 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:25.070902    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:25.071102 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:25.071006    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:25.175853 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:25.175822    2573 generic.go:356] "Generic (PLEG): container finished" podID="203b8b56-b634-4a8b-85b1-46aae63f6dd8" containerID="db257033c0c4f9aeab92cb24613a828e0657b9625b9ae5eef6ca99f573a608eb" exitCode=0
Oct 31 18:20:25.176302 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:25.175907    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpvmh" event={"ID":"203b8b56-b634-4a8b-85b1-46aae63f6dd8","Type":"ContainerDied","Data":"db257033c0c4f9aeab92cb24613a828e0657b9625b9ae5eef6ca99f573a608eb"}
Oct 31 18:20:25.178558 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:25.178514    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6" event={"ID":"8c12bcbd-bf78-49c1-b60c-0c39e16faf61","Type":"ContainerStarted","Data":"a9bfe4d943918ac8a9c1f894db6180e2bd379457f85f7553e998255a27b4b754"}
Oct 31 18:20:25.180153 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:25.180117    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tjxmm" event={"ID":"32e9756e-22b8-4adb-a36f-dc837581e01a","Type":"ContainerStarted","Data":"7d87ea89e36ad0eaf3cb735cb2109b56932dfc30aa7114e243d3bd85eb11e662"}
Oct 31 18:20:25.414045 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:25.413946    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tjxmm" podStartSLOduration=3.105433652 podStartE2EDuration="11.413929199s" totalImagesPullingTime="8.308495547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:25.41344269 +0000 UTC m=+11.791560990" watchObservedRunningTime="2026-10-31 18:20:25.413929199 +0000 UTC m=+11.792047497"
Oct 31 18:20:26.185054 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:26.185025    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rjntd" event={"ID":"207d8f7f-20e9-430d-a74d-42419bc4da86","Type":"ContainerStarted","Data":"8c7c99c42047a9f64d54b0f5afd4e34fea299e41d364615efb672edc9670a45e"}
Oct 31 18:20:26.297878 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:26.297825    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rjntd" podStartSLOduration=3.850800082 podStartE2EDuration="12.297811254s" totalImagesPullingTime="8.447011172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:26.296995612 +0000 UTC m=+12.675113910" watchObservedRunningTime="2026-10-31 18:20:26.297811254 +0000 UTC m=+12.675929551"
Oct 31 18:20:27.070900 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:27.070866    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:27.071041 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:27.070877    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:27.071041 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:27.070917    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:27.071041 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:27.070995    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:27.071186 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:27.071111    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:27.071224 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:27.071200    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:27.189184 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:27.189149    2573 generic.go:356] "Generic (PLEG): container finished" podID="203b8b56-b634-4a8b-85b1-46aae63f6dd8" containerID="8fd79eff7a36c2af19b51d5da1207ad190fe174f88e5939e1b4934a82f5a8d9c" exitCode=0
Oct 31 18:20:27.189890 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:27.189233    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpvmh" event={"ID":"203b8b56-b634-4a8b-85b1-46aae63f6dd8","Type":"ContainerDied","Data":"8fd79eff7a36c2af19b51d5da1207ad190fe174f88e5939e1b4934a82f5a8d9c"}
Oct 31 18:20:27.191735 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:27.191704    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6" event={"ID":"8c12bcbd-bf78-49c1-b60c-0c39e16faf61","Type":"ContainerStarted","Data":"28d6986c6985cabf9d18ba7c26b3350f31ba251574f54351f7c7d0d538a88354"}
Oct 31 18:20:27.193355 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:27.193333    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gpjxn" event={"ID":"828b1043-03ed-45a2-afb3-160da3dce8c0","Type":"ContainerStarted","Data":"f2d4119dfa02d130ea3d94e4f44275079a98b382c6c8e65a8bf209e67bdc940c"}
Oct 31 18:20:27.480113 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:27.480013    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6" podStartSLOduration=4.129411381 podStartE2EDuration="13.479998055s" totalImagesPullingTime="9.350586674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:27.47970166 +0000 UTC m=+13.857819959" watchObservedRunningTime="2026-10-31 18:20:27.479998055 +0000 UTC m=+13.858116352"
Oct 31 18:20:27.594778 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:27.594713    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gpjxn" podStartSLOduration=3.662667206 podStartE2EDuration="13.594695622s" totalImagesPullingTime="9.932028416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:27.594454203 +0000 UTC m=+13.972572500" watchObservedRunningTime="2026-10-31 18:20:27.594695622 +0000 UTC m=+13.972813921"
Oct 31 18:20:29.070049 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:29.070013    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:29.070639 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:29.070055    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:29.070639 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:29.070054    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:29.070639 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:29.070148    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:29.070639 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:29.070229    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:29.070639 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:29.070306    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:29.200567 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:29.200534    2573 generic.go:356] "Generic (PLEG): container finished" podID="203b8b56-b634-4a8b-85b1-46aae63f6dd8" containerID="456e16070e2e6f7b2cc0471ab07e2750eb3dbdf1ce7abb5f2231691ad9e6cf20" exitCode=0
Oct 31 18:20:29.200736 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:29.200593    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpvmh" event={"ID":"203b8b56-b634-4a8b-85b1-46aae63f6dd8","Type":"ContainerDied","Data":"456e16070e2e6f7b2cc0471ab07e2750eb3dbdf1ce7abb5f2231691ad9e6cf20"}
Oct 31 18:20:30.961537 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:30.961357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:30.961903 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:30.961581    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:30.961903 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:30.961519    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:30.961903 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:30.961651    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs podName:ac38de34-5c48-430d-b06f-7015c64fbb56 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.961637921 +0000 UTC m=+33.339756197 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs") pod "network-metrics-daemon-r92q2" (UID: "ac38de34-5c48-430d-b06f-7015c64fbb56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 18:20:30.961903 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:30.961687    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:30.961903 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:30.961743    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret podName:c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.961727816 +0000 UTC m=+33.339846098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret") pod "global-pull-secret-syncer-h72m5" (UID: "c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7") : object "kube-system"/"original-pull-secret" not registered
Oct 31 18:20:31.062608 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.062562    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68bd\" (UniqueName: \"kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd\") pod \"network-check-target-9mlcf\" (UID: \"b4b27f4e-abf1-407d-a33d-90e23a5fc861\") " pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:31.062698 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:31.062685    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 18:20:31.062732 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:31.062698    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 18:20:31.062732 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:31.062706    2573 projected.go:196] Error preparing data for projected volume kube-api-access-r68bd for pod openshift-network-diagnostics/network-check-target-9mlcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:31.062812 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:31.062756    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd podName:b4b27f4e-abf1-407d-a33d-90e23a5fc861 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:47.062739033 +0000 UTC m=+33.440857323 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r68bd" (UniqueName: "kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd") pod "network-check-target-9mlcf" (UID: "b4b27f4e-abf1-407d-a33d-90e23a5fc861") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 18:20:31.069828 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.069802    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:31.069936 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:31.069917    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:31.069996 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.069985    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:31.070037 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.070008    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:31.070088 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:31.070072    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:31.070164 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:31.070147    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:31.207991 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.207969    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:20:31.208355 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.208332    2573 generic.go:356] "Generic (PLEG): container finished" podID="0a79a203-4e31-41e5-af54-c8d07320badd" containerID="7c18a3c4daf52914848e65ff05ea03367b5189175fc8e34ebc92757192cf978e" exitCode=1
Oct 31 18:20:31.208413 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.208371    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn" event={"ID":"0a79a203-4e31-41e5-af54-c8d07320badd","Type":"ContainerStarted","Data":"e06be749b2435b5dcac8f451ea1b559d7caa2a1c395d2e02231548a3d4300e1c"}
Oct 31 18:20:31.208449 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.208423    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn" event={"ID":"0a79a203-4e31-41e5-af54-c8d07320badd","Type":"ContainerStarted","Data":"d3e3a174e53ff97c3c6e327fdb00e4c3cba187b36ef1b3ad0901cafaf5f1b8d6"}
Oct 31 18:20:31.208500 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.208453    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn" event={"ID":"0a79a203-4e31-41e5-af54-c8d07320badd","Type":"ContainerDied","Data":"7c18a3c4daf52914848e65ff05ea03367b5189175fc8e34ebc92757192cf978e"}
Oct 31 18:20:31.208538 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.208508    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn" event={"ID":"0a79a203-4e31-41e5-af54-c8d07320badd","Type":"ContainerStarted","Data":"f719020436ed410eeed7d62192a4a9df6ede79499081e7791c76d25b9b1a5192"}
Oct 31 18:20:31.499912 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.499754    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:31.500078 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.500007    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 18:20:31.500641 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:31.500610    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:32.212625 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:32.212592    2573 generic.go:356] "Generic (PLEG): container finished" podID="203b8b56-b634-4a8b-85b1-46aae63f6dd8" containerID="0e3ad3389f416863c1d8dc6dde1c328b9c8818f1ec73c95c75e2cc62fbb40982" exitCode=0
Oct 31 18:20:32.213074 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:32.212674    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpvmh" event={"ID":"203b8b56-b634-4a8b-85b1-46aae63f6dd8","Type":"ContainerDied","Data":"0e3ad3389f416863c1d8dc6dde1c328b9c8818f1ec73c95c75e2cc62fbb40982"}
Oct 31 18:20:32.215510 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:32.215492    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:20:32.215828 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:32.215808    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn" event={"ID":"0a79a203-4e31-41e5-af54-c8d07320badd","Type":"ContainerStarted","Data":"114c9b47e769345a7a452f1ce3f2727be1ff1fee140ebeb9ee38b9ab5215a852"}
Oct 31 18:20:32.215929 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:32.215862    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn" event={"ID":"0a79a203-4e31-41e5-af54-c8d07320badd","Type":"ContainerStarted","Data":"5e9cd2aff3cc5cba348c1abc5bebedce2f3a7b3e80608f5cc26476f8f4755100"}
Oct 31 18:20:33.070431 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:33.070400    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:33.070431 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:33.070430    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:33.070654 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:33.070517    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:33.070654 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:33.070554    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:33.070654 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:33.070624    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:33.070742 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:33.070691    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:33.220673 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:33.220646    2573 generic.go:356] "Generic (PLEG): container finished" podID="203b8b56-b634-4a8b-85b1-46aae63f6dd8" containerID="6f0a8d9755332d7eff87318f35f238797cfe9f4c8289d7c5e0401d584fbff4a4" exitCode=0
Oct 31 18:20:33.220673 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:33.220676    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpvmh" event={"ID":"203b8b56-b634-4a8b-85b1-46aae63f6dd8","Type":"ContainerDied","Data":"6f0a8d9755332d7eff87318f35f238797cfe9f4c8289d7c5e0401d584fbff4a4"}
Oct 31 18:20:34.224938 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:34.224914    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:20:34.225402 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:34.225275    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn" event={"ID":"0a79a203-4e31-41e5-af54-c8d07320badd","Type":"ContainerStarted","Data":"60af036b859cfc3215107f71270222bb2157758f77976f9066b3dd08a6cf488d"}
Oct 31 18:20:34.228127 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:34.228102    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpvmh" event={"ID":"203b8b56-b634-4a8b-85b1-46aae63f6dd8","Type":"ContainerStarted","Data":"2ce300bda754e9e1a07eabafe41c11a9d6692f531227a47a3634bcecbf8d3d04"}
Oct 31 18:20:34.351783 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:34.351735    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xpvmh" podStartSLOduration=8.057404354 podStartE2EDuration="20.351722449s" totalImagesPullingTime="12.294318095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:34.351352868 +0000 UTC m=+20.729471164" watchObservedRunningTime="2026-10-31 18:20:34.351722449 +0000 UTC m=+20.729840746"
Oct 31 18:20:35.070669 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:35.070636    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:35.070669 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:35.070653    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:35.070910 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:35.070734    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:35.070910 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:35.070737    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:35.070910 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:35.070804    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:35.070910 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:35.070866    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:36.235214 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:36.235189    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:20:36.235998 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:36.235563    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn" event={"ID":"0a79a203-4e31-41e5-af54-c8d07320badd","Type":"ContainerStarted","Data":"460ffb59fc3a45c36f034c98cbf3f9a6c99fed6d07ff69f4fd76f5ddbfd3c159"}
Oct 31 18:20:36.235998 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:36.235832    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:36.235998 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:36.235890    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:36.236128 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:36.236005    2573 scope.go:122] "RemoveContainer" containerID="7c18a3c4daf52914848e65ff05ea03367b5189175fc8e34ebc92757192cf978e"
Oct 31 18:20:36.252160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:36.252137    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:36.252249 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:36.252234    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:37.070103 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:37.070066    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:37.070436 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:37.070124    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:37.070436 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:37.070087    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:37.070436 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:37.070235    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:37.070436 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:37.070297    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:37.070436 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:37.070376    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:37.241570 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:37.241545    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:20:37.241973 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:37.241895    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn" event={"ID":"0a79a203-4e31-41e5-af54-c8d07320badd","Type":"ContainerStarted","Data":"9f0e724dbc4749df3a94cb87f2561278d13cbfd09a60b41a741a9ae949bd52d7"}
Oct 31 18:20:37.242016 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:37.241994    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 18:20:37.362985 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:37.362904    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn" podStartSLOduration=8.188519313 podStartE2EDuration="23.362890687s" totalImagesPullingTime="15.174371374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 18:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:37.362663985 +0000 UTC m=+23.740782282" watchObservedRunningTime="2026-10-31 18:20:37.362890687 +0000 UTC m=+23.741008974"
Oct 31 18:20:38.077654 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:38.077396    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h72m5"]
Oct 31 18:20:38.077835 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:38.077654    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:38.077835 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:38.077766    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:38.078046 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:38.078021    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r92q2"]
Oct 31 18:20:38.078132 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:38.078118    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:38.078247 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:38.078226    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:38.078609 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:38.078589    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9mlcf"]
Oct 31 18:20:38.078695 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:38.078655    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:38.078730 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:38.078712    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:38.243604 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:38.243578    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 18:20:39.685098 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:39.685071    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:40.070173 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:40.070133    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:40.070173 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:40.070156    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:40.070419 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:40.070225    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:40.070419 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:40.070240    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:40.070419 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:40.070299    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:40.070419 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:40.070386    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:42.069914 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:42.069832    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:42.069914 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:42.069846    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:42.069914 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:42.069832    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:42.070324 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:42.069940    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h72m5" podUID="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7"
Oct 31 18:20:42.070324 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:42.069981    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mlcf" podUID="b4b27f4e-abf1-407d-a33d-90e23a5fc861"
Oct 31 18:20:42.070324 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:42.070051    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r92q2" podUID="ac38de34-5c48-430d-b06f-7015c64fbb56"
Oct 31 18:20:43.913912 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:43.913801    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:43.945509 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:43.945463    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeReady"
Oct 31 18:20:43.945655 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:43.945611    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 18:20:44.062568 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.062543    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-glvbh"]
Oct 31 18:20:44.066984 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.066963    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.068433 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.068414    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wqsdj"]
Oct 31 18:20:44.072019 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.072002    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wqsdj"
Oct 31 18:20:44.075615 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.075594    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:44.075764 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.075749    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:44.075815 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.075801    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:44.129414 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.129388    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 18:20:44.129414 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.129397    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-glvbh"]
Oct 31 18:20:44.129676 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.129528    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 18:20:44.129676 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.129541    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ms9mk\""
Oct 31 18:20:44.129676 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.129581    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5t2xq\""
Oct 31 18:20:44.129676 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.129594    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 18:20:44.129676 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.129582    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 18:20:44.129676 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.129622    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 18:20:44.149383 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.149356    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wqsdj"]
Oct 31 18:20:44.155815 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.155792    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d647747-ac4a-41c4-bec2-f9aff836f5bb-config-volume\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.155930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.155832    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d647747-ac4a-41c4-bec2-f9aff836f5bb-tmp-dir\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.155930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.155876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8771b315-47d1-4bd6-b8f7-cc1cb548a789-cert\") pod \"ingress-canary-wqsdj\" (UID: \"8771b315-47d1-4bd6-b8f7-cc1cb548a789\") " pod="openshift-ingress-canary/ingress-canary-wqsdj"
Oct 31 18:20:44.156043 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.156022    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.156095 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.156057    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-255ph\" (UniqueName: \"kubernetes.io/projected/9d647747-ac4a-41c4-bec2-f9aff836f5bb-kube-api-access-255ph\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.156128 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.156109    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlltr\" (UniqueName: \"kubernetes.io/projected/8771b315-47d1-4bd6-b8f7-cc1cb548a789-kube-api-access-hlltr\") pod \"ingress-canary-wqsdj\" (UID: \"8771b315-47d1-4bd6-b8f7-cc1cb548a789\") " pod="openshift-ingress-canary/ingress-canary-wqsdj"
Oct 31 18:20:44.187855 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.187796    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:44.229645 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.229622    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 18:20:44.229768 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.229748    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 18:20:44.229822 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.229800    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 18:20:44.229930 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.229910    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 18:20:44.230012 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.229999    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vv64h\""
Oct 31 18:20:44.230056 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.230005    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-77xvx\""
Oct 31 18:20:44.230103 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.230093    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7mm2n\""
Oct 31 18:20:44.256511 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.256468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d647747-ac4a-41c4-bec2-f9aff836f5bb-tmp-dir\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.256618 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.256547    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8771b315-47d1-4bd6-b8f7-cc1cb548a789-cert\") pod \"ingress-canary-wqsdj\" (UID: \"8771b315-47d1-4bd6-b8f7-cc1cb548a789\") " pod="openshift-ingress-canary/ingress-canary-wqsdj"
Oct 31 18:20:44.256618 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.256574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.256618 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.256589    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255ph\" (UniqueName: \"kubernetes.io/projected/9d647747-ac4a-41c4-bec2-f9aff836f5bb-kube-api-access-255ph\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.256765 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.256657    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlltr\" (UniqueName: \"kubernetes.io/projected/8771b315-47d1-4bd6-b8f7-cc1cb548a789-kube-api-access-hlltr\") pod \"ingress-canary-wqsdj\" (UID: \"8771b315-47d1-4bd6-b8f7-cc1cb548a789\") " pod="openshift-ingress-canary/ingress-canary-wqsdj"
Oct 31 18:20:44.256765 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:44.256684    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 18:20:44.256765 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.256714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d647747-ac4a-41c4-bec2-f9aff836f5bb-config-volume\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.256765 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:44.256729    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls podName:9d647747-ac4a-41c4-bec2-f9aff836f5bb nodeName:}" failed. No retries permitted until 2026-10-31 18:20:44.756714696 +0000 UTC m=+31.134832995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls") pod "dns-default-glvbh" (UID: "9d647747-ac4a-41c4-bec2-f9aff836f5bb") : secret "dns-default-metrics-tls" not found
Oct 31 18:20:44.256765 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.256749    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d647747-ac4a-41c4-bec2-f9aff836f5bb-tmp-dir\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.257222 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.257203    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d647747-ac4a-41c4-bec2-f9aff836f5bb-config-volume\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.260329 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.260306    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8771b315-47d1-4bd6-b8f7-cc1cb548a789-cert\") pod \"ingress-canary-wqsdj\" (UID: \"8771b315-47d1-4bd6-b8f7-cc1cb548a789\") " pod="openshift-ingress-canary/ingress-canary-wqsdj"
Oct 31 18:20:44.314429 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.314403    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255ph\" (UniqueName: \"kubernetes.io/projected/9d647747-ac4a-41c4-bec2-f9aff836f5bb-kube-api-access-255ph\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.314580 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.314435    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlltr\" (UniqueName: \"kubernetes.io/projected/8771b315-47d1-4bd6-b8f7-cc1cb548a789-kube-api-access-hlltr\") pod \"ingress-canary-wqsdj\" (UID: \"8771b315-47d1-4bd6-b8f7-cc1cb548a789\") " pod="openshift-ingress-canary/ingress-canary-wqsdj"
Oct 31 18:20:44.386795 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.386762    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wqsdj"
Oct 31 18:20:44.549266 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.549224    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:44.700628 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.700556    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:44.760324 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.760297    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:44.760460 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:44.760444    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 18:20:44.760532 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:44.760521    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls podName:9d647747-ac4a-41c4-bec2-f9aff836f5bb nodeName:}" failed. No retries permitted until 2026-10-31 18:20:45.760504483 +0000 UTC m=+32.138622776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls") pod "dns-default-glvbh" (UID: "9d647747-ac4a-41c4-bec2-f9aff836f5bb") : secret "dns-default-metrics-tls" not found
Oct 31 18:20:44.770402 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:44.769081    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wqsdj"]
Oct 31 18:20:44.786808 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:44.786783    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8771b315_47d1_4bd6_b8f7_cc1cb548a789.slice/crio-950970b6b956cff6fe7470fa50d663e2dfef38208039af1746ecf60b9094cf9d WatchSource:0}: Error finding container 950970b6b956cff6fe7470fa50d663e2dfef38208039af1746ecf60b9094cf9d: Status 404 returned error can't find the container with id 950970b6b956cff6fe7470fa50d663e2dfef38208039af1746ecf60b9094cf9d
Oct 31 18:20:45.260225 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.260182    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wqsdj" event={"ID":"8771b315-47d1-4bd6-b8f7-cc1cb548a789","Type":"ContainerStarted","Data":"950970b6b956cff6fe7470fa50d663e2dfef38208039af1746ecf60b9094cf9d"}
Oct 31 18:20:45.429738 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.429692    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gwg8k"]
Oct 31 18:20:45.470406 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.470378    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.521073 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.521002    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gwg8k"]
Oct 31 18:20:45.566199 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.566168    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-data-volume\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.566323 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.566203    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.566323 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.566237    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.566393 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.566322    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-crio-socket\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.566393 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.566369    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km59v\" (UniqueName: \"kubernetes.io/projected/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-api-access-km59v\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.597160 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.597135    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 18:20:45.597290 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.597209    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 18:20:45.597290 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.597211    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 18:20:45.597400 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.597357    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nkjx2\""
Oct 31 18:20:45.597400 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.597366    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 18:20:45.665940 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.665912    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:45.666808 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.666793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-data-volume\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.666869 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.666816    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.666869 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.666839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.666869 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.666860    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-crio-socket\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.667001 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.666943    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km59v\" (UniqueName: \"kubernetes.io/projected/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-api-access-km59v\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.667001 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:45.666957    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:45.667092 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:45.667009    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.166992455 +0000 UTC m=+32.545110732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:45.667092 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:45.667029    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 18:20:45.667092 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.667032    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-crio-socket\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.667092 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:45.667069    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:46.167058831 +0000 UTC m=+32.545177108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : configmap "kube-rbac-proxy" not found
Oct 31 18:20:45.667285 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.667148    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-data-volume\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.668438 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.668419    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-vlnmf-hb4xw"]
Oct 31 18:20:45.687415 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.687398    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:45.723791 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.723762    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km59v\" (UniqueName: \"kubernetes.io/projected/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-api-access-km59v\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:45.727713 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.727691    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vlnmf-hb4xw"]
Oct 31 18:20:45.739454 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.739427    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 18:20:45.739668 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.739654    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6wm76\""
Oct 31 18:20:45.768153 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.768129    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-archives-path\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:45.768256 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.768182    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:45.768318 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.768262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v2s4\" (UniqueName: \"kubernetes.io/projected/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-kube-api-access-2v2s4\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:45.768318 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:45.768278    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 18:20:45.768318 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.768292    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-service-ca-bundle\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:45.768318 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.768316    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-trusted-ca-bundle\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:45.768512 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:45.768329    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls podName:9d647747-ac4a-41c4-bec2-f9aff836f5bb nodeName:}" failed. No retries permitted until 2026-10-31 18:20:47.768313696 +0000 UTC m=+34.146431986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls") pod "dns-default-glvbh" (UID: "9d647747-ac4a-41c4-bec2-f9aff836f5bb") : secret "dns-default-metrics-tls" not found
Oct 31 18:20:45.789945 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.789880    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 18:20:45.869243 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.869203    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-archives-path\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:45.869381 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.869265    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v2s4\" (UniqueName: \"kubernetes.io/projected/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-kube-api-access-2v2s4\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:45.869438 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.869421    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-service-ca-bundle\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:45.869493 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.869459    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-trusted-ca-bundle\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:45.869653 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.869638    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-archives-path\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:45.869936 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.869922    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-service-ca-bundle\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:45.870139 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.870123    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-trusted-ca-bundle\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:45.925796 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:45.925766    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v2s4\" (UniqueName: \"kubernetes.io/projected/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-kube-api-access-2v2s4\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:46.012333 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:46.012304    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:46.172234 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:46.172198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:46.172234 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:46.172239    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:46.172458 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:46.172331    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 18:20:46.172458 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:46.172342    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:46.172458 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:46.172401    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:47.172383173 +0000 UTC m=+33.550501452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : configmap "kube-rbac-proxy" not found
Oct 31 18:20:46.172458 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:46.172418    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:47.172410412 +0000 UTC m=+33.550528688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:46.417357 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:46.417275    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vlnmf-hb4xw"]
Oct 31 18:20:46.420812 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:46.420771    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5fb89e_04c5_4f69_ac9b_a12e9e14d657.slice/crio-91b208cc2de4062418d3cebdc99d47d8a42f3fb382db917c2949fd2df9f0c523 WatchSource:0}: Error finding container 91b208cc2de4062418d3cebdc99d47d8a42f3fb382db917c2949fd2df9f0c523: Status 404 returned error can't find the container with id 91b208cc2de4062418d3cebdc99d47d8a42f3fb382db917c2949fd2df9f0c523
Oct 31 18:20:46.978804 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:46.978768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:46.978981 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:46.978877    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:46.978981 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:46.978950    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 18:20:46.979076 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:46.979021    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs podName:ac38de34-5c48-430d-b06f-7015c64fbb56 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:18.979001586 +0000 UTC m=+65.357119866 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs") pod "network-metrics-daemon-r92q2" (UID: "ac38de34-5c48-430d-b06f-7015c64fbb56") : secret "metrics-daemon-secret" not found
Oct 31 18:20:46.981423 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:46.981397    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:47.080198 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.080164    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68bd\" (UniqueName: \"kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd\") pod \"network-check-target-9mlcf\" (UID: \"b4b27f4e-abf1-407d-a33d-90e23a5fc861\") " pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:47.082939 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.082918    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68bd\" (UniqueName: \"kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd\") pod \"network-check-target-9mlcf\" (UID: \"b4b27f4e-abf1-407d-a33d-90e23a5fc861\") " pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:47.092799 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.092768    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:47.098942 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.098924    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:47.181127 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.181097    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:47.181254 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.181174    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:47.181495 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:47.181282    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 18:20:47.181495 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:47.181391    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:49.181371822 +0000 UTC m=+35.559490129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : configmap "kube-rbac-proxy" not found
Oct 31 18:20:47.181495 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:47.181289    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:47.181495 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:47.181462    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:49.181443571 +0000 UTC m=+35.559561866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:47.267022 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.266985    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vlnmf-hb4xw" event={"ID":"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657","Type":"ContainerStarted","Data":"91b208cc2de4062418d3cebdc99d47d8a42f3fb382db917c2949fd2df9f0c523"}
Oct 31 18:20:47.428317 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.428265    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:47.513661 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.513624    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h72m5"]
Oct 31 18:20:47.517702 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:47.517648    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d62e4b_14f2_4ae5_83e4_bddf1230ddd7.slice/crio-50d556d2fdfe1e392c5679657ad6ba99397cd7bfb673512017eac9b75e0ba4be WatchSource:0}: Error finding container 50d556d2fdfe1e392c5679657ad6ba99397cd7bfb673512017eac9b75e0ba4be: Status 404 returned error can't find the container with id 50d556d2fdfe1e392c5679657ad6ba99397cd7bfb673512017eac9b75e0ba4be
Oct 31 18:20:47.562516 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.562487    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9mlcf"]
Oct 31 18:20:47.591189 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.591156    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:47.766057 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.766024    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:47.786338 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:47.786264    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:47.786486 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:47.786435    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 18:20:47.786559 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:47.786540    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls podName:9d647747-ac4a-41c4-bec2-f9aff836f5bb nodeName:}" failed. No retries permitted until 2026-10-31 18:20:51.786517594 +0000 UTC m=+38.164635899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls") pod "dns-default-glvbh" (UID: "9d647747-ac4a-41c4-bec2-f9aff836f5bb") : secret "dns-default-metrics-tls" not found
Oct 31 18:20:48.270666 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.270633    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9mlcf" event={"ID":"b4b27f4e-abf1-407d-a33d-90e23a5fc861","Type":"ContainerStarted","Data":"364a6458c6c86a01d0cbd92d74698ee653a5b0df3305828cf44eaac37231b16e"}
Oct 31 18:20:48.271916 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.271886    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h72m5" event={"ID":"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7","Type":"ContainerStarted","Data":"50d556d2fdfe1e392c5679657ad6ba99397cd7bfb673512017eac9b75e0ba4be"}
Oct 31 18:20:48.566641 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.566611    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qxdz6"]
Oct 31 18:20:48.586035 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.586007    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:48.633860 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.633833    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qxdz6"]
Oct 31 18:20:48.682609 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.682578    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 18:20:48.682609 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.682578    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 18:20:48.682824 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.682669    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ghldp\""
Oct 31 18:20:48.689095 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.688902    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 18:20:48.689095 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.688975    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 18:20:48.693609 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.693341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-signing-key\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:48.693609 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.693449    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbdvb\" (UniqueName: \"kubernetes.io/projected/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-kube-api-access-wbdvb\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:48.693609 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.693503    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-signing-cabundle\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:48.794753 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.794664    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbdvb\" (UniqueName: \"kubernetes.io/projected/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-kube-api-access-wbdvb\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:48.794914 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.794778    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-signing-cabundle\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:48.794914 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.794830    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-signing-key\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:48.796651 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.796611    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:48.797386 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.797342    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-signing-cabundle\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:48.799287 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.799264    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-signing-key\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:48.856396 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.856335    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbdvb\" (UniqueName: \"kubernetes.io/projected/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-kube-api-access-wbdvb\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:48.917059 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.917026    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:48.957759 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:48.957727    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:49.198654 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:49.198582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:49.198808 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:49.198702    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:49.198808 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:49.198753    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:49.198808 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:49.198793    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 18:20:49.198956 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:49.198829    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:53.198815154 +0000 UTC m=+39.576933452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:49.198956 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:49.198847    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:53.1988393 +0000 UTC m=+39.576957575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : configmap "kube-rbac-proxy" not found
Oct 31 18:20:49.276588 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:49.276551    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wqsdj" event={"ID":"8771b315-47d1-4bd6-b8f7-cc1cb548a789","Type":"ContainerStarted","Data":"4929744e4f4b8dd18a186f8bfe28c0c2763c72d7fb483a2988d0a73133a7a6a2"}
Oct 31 18:20:49.343326 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:49.343299    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qxdz6"]
Oct 31 18:20:49.346687 ip-10-0-142-120 kubenswrapper[2573]: W1031 18:20:49.346657    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fb4572_e09f_4de5_b9ba_b7bbd4a306c8.slice/crio-6ccdc557353dfd74e57c146fd918fa6b78a570df7ebb22a051ce50dfea83b014 WatchSource:0}: Error finding container 6ccdc557353dfd74e57c146fd918fa6b78a570df7ebb22a051ce50dfea83b014: Status 404 returned error can't find the container with id 6ccdc557353dfd74e57c146fd918fa6b78a570df7ebb22a051ce50dfea83b014
Oct 31 18:20:49.391787 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:49.391738    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wqsdj" podStartSLOduration=1.65964418 podStartE2EDuration="5.391720706s" totalImagesPullingTime="3.732076526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:49.39135977 +0000 UTC m=+35.769478081" watchObservedRunningTime="2026-10-31 18:20:49.391720706 +0000 UTC m=+35.769839003"
Oct 31 18:20:49.490107 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:49.490076    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:49.708550 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:49.708466    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:49.911268 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:49.911184    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:50.138671 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:50.138623    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:50.282701 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:50.282510    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6" event={"ID":"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8","Type":"ContainerStarted","Data":"6ccdc557353dfd74e57c146fd918fa6b78a570df7ebb22a051ce50dfea83b014"}
Oct 31 18:20:50.284365 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:50.284323    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vlnmf-hb4xw" event={"ID":"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657","Type":"ContainerStarted","Data":"b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b"}
Oct 31 18:20:50.305757 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:50.305734    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:50.400894 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:50.400830    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-vlnmf-hb4xw" podStartSLOduration=2.5073209309999998 podStartE2EDuration="5.400812789s" totalImagesPullingTime="2.893491858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:50.40021064 +0000 UTC m=+36.778328936" watchObservedRunningTime="2026-10-31 18:20:50.400812789 +0000 UTC m=+36.778931087"
Oct 31 18:20:51.823734 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:51.823602    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:51.823734 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:51.823730    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 18:20:51.824190 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:51.823808    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls podName:9d647747-ac4a-41c4-bec2-f9aff836f5bb nodeName:}" failed. No retries permitted until 2026-10-31 18:20:59.823788104 +0000 UTC m=+46.201906383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls") pod "dns-default-glvbh" (UID: "9d647747-ac4a-41c4-bec2-f9aff836f5bb") : secret "dns-default-metrics-tls" not found
Oct 31 18:20:52.291730 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:52.291686    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9mlcf" event={"ID":"b4b27f4e-abf1-407d-a33d-90e23a5fc861","Type":"ContainerStarted","Data":"3aa85b7291735736c23fb0c3fd9c8d21c7bc457816a630a92170daae674e262e"}
Oct 31 18:20:52.292443 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:52.292411    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:52.406706 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:52.406603    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9mlcf" podStartSLOduration=33.845055155 podStartE2EDuration="38.406585008s" totalImagesPullingTime="4.561529853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:52.405565699 +0000 UTC m=+38.783683996" watchObservedRunningTime="2026-10-31 18:20:52.406585008 +0000 UTC m=+38.784703305"
Oct 31 18:20:53.076443 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:53.076359    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:53.235744 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:53.235711    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:53.235744 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:53.235753    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:53.235954 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:53.235864    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:53.235954 ip-10-0-142-120 kubenswrapper[2573]: E1031 18:20:53.235915    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:01.235903395 +0000 UTC m=+47.614021684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:53.236358 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:53.236337    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:53.295668 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:53.295630    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6" event={"ID":"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8","Type":"ContainerStarted","Data":"98d8978b4f360ea50821f5af999d1711598bc69235baf63dac9ab49a4dcd24bd"}
Oct 31 18:20:53.320431 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:53.320405    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tjxmm_32e9756e-22b8-4adb-a36f-dc837581e01a/dns-node-resolver/0.log"
Oct 31 18:20:53.408438 ip-10-0-142-120 kubenswrapper[2573]: I1031 18:20:53.408349    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6" podStartSLOduration=1.994604087 podStartE2EDuration="5.408335287s" totalImagesPullingTime="3.4137312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:20:53.408160818 +0000 UTC m=+39.786279113" watchObservedRunningTime="2026-10-31 18:20:53.408335287 +0000 UTC m=+39.786453584"
Oct 31 18:20:53.779718 ip-10-0-142-120 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 18:20:53.795864 ip-10-0-142-120 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 18:20:53.796038 ip-10-0-142-120 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 18:20:53.796085 ip-10-0-142-120 systemd[1]: kubelet.service: Consumed 2.341s CPU time.
Oct 31 18:20:53.813136 ip-10-0-142-120 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 18:20:53.909504 ip-10-0-142-120 kubenswrapper[4484]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 18:20:53.909504 ip-10-0-142-120 kubenswrapper[4484]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 18:20:53.909504 ip-10-0-142-120 kubenswrapper[4484]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 18:20:53.910029 ip-10-0-142-120 kubenswrapper[4484]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 18:20:53.911527 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911429    4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:53.911527 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911526    4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911530    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911533    4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911536    4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911539    4484 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911541    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911544    4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911546    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911549    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911551    4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911554    4484 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911557    4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911559    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911562    4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911565    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911567    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911570    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911578    4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911580    4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:53.911600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911583    4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911585    4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911588    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911590    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911593    4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911595    4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911597    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911601    4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911603    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911606    4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911609    4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911614    4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911618    4484 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911621    4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911624    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911626    4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911635    4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911637    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911640    4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:53.912059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911642    4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911644    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911647    4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911649    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911652    4484 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911654    4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911657    4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911659    4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911661    4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911664    4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911666    4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911668    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911671    4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911674    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911677    4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911680    4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911682    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911684    4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911687    4484 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911689    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:53.912612 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911692    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911694    4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911698    4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911700    4484 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911703    4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911705    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911708    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911710    4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911713    4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911716    4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911723    4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911726    4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911728    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911731    4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911734    4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911736    4484 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911739    4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911741    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911744    4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911746    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:53.913237 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911748    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911751    4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911754    4484 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911756    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911759    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911761    4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911764    4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911766    4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911769    4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911772    4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911774    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911777    4484 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911779    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911782    4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911784    4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911786    4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911812    4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911815    4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911820    4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:53.913749 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911824    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911827    4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911830    4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911833    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911841    4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911843    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911846    4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.911848    4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912741    4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912751    4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912757    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912761    4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912765    4484 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912769    4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912773    4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912778    4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912782    4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912786    4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912791    4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:53.914243 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912795    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912799    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912803    4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912808    4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912812    4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912817    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912820    4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912824    4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912829    4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912832    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912834    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912836    4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912839    4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912843    4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912848    4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912851    4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912854    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912856    4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912866    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912869    4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:53.914996 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912871    4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912874    4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912876    4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912879    4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912881    4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912883    4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912886    4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912890    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912893    4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912896    4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912899    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912901    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912904    4484 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912906    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912909    4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912911    4484 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912914    4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912916    4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912919    4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:53.915662 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912921    4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912924    4484 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912926    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912928    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912930    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912933    4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912936    4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912938    4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912941    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912944    4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912946    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912949    4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912956    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912959    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912961    4484 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912964    4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912967    4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912969    4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912971    4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912974    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:53.916160 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912976    4484 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912979    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912983    4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912985    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912988    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912992    4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912996    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.912999    4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913001    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913004    4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913006    4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913008    4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913011    4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913013    4484 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913015    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913018    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913020    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913023    4484 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913025    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:53.916657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913028    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913031    4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913033    4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913036    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913038    4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913041    4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913048    4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913051    4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913053    4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913056    4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913060    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913063    4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913065    4484 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913067    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913070    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913072    4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913080    4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913710    4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913718    4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913721    4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:53.917107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913724    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913726    4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913729    4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913732    4484 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913734    4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913737    4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913739    4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913742    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913744    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913747    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913749    4484 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913752    4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913754    4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913757    4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913760    4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913763    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913765    4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913767    4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913771    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913773    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:53.917626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913776    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913778    4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913781    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913783    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913785    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913788    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913790    4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913793    4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913795    4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913797    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913800    4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913802    4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913804    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913806    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913809    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913811    4484 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913815    4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913818    4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913822    4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:53.918140 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913824    4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913826    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913829    4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913831    4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913834    4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913838    4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913840    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913842    4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913845    4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913847    4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913849    4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913854    4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913857    4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913859    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913861    4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913864    4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913866    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913869    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913871    4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:53.918600 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913873    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913876    4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913878    4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913880    4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913883    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913885    4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913887    4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913889    4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913893    4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913896    4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913899    4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913902    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913904    4484 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913907    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913909    4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913912    4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913914    4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913917    4484 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913919    4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913922    4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:53.919057 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913924    4484 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913927    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913929    4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913932    4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913934    4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913937    4484 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913940    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913942    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913945    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913947    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913951    4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913953    4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913956    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913958    4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913961    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913963    4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913965    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913967    4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913970    4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:53.919536 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913972    4484 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913974    4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913977    4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913979    4484 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913981    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.913987    4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914293    4484 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914313    4484 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914368    4484 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914375    4484 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914382    4484 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914387    4484 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914393    4484 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914399    4484 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914404    4484 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914414    4484 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914420    4484 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914425    4484 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914430    4484 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914435    4484 flags.go:64] FLAG: --cgroup-root=""
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914441    4484 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914446    4484 flags.go:64] FLAG: --client-ca-file=""
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914451    4484 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 18:20:53.920098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914461    4484 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914466    4484 flags.go:64] FLAG: --cluster-domain=""
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914493    4484 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914499    4484 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914504    4484 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914509    4484 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914515    4484 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914520    4484 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914525    4484 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914534    4484 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914539    4484 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914544    4484 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914548    4484 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914553    4484 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914561    4484 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914567    4484 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914572    4484 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914580    4484 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914585    4484 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914590    4484 flags.go:64] FLAG: --enable-server="true"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914594    4484 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914602    4484 flags.go:64] FLAG: --event-burst="100"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914607    4484 flags.go:64] FLAG: --event-qps="50"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914611    4484 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 18:20:53.920660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914616    4484 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914620    4484 flags.go:64] FLAG: --eviction-hard=""
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914631    4484 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914636    4484 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914640    4484 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914645    4484 flags.go:64] FLAG: --eviction-soft=""
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914651    4484 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914656    4484 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914661    4484 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914665    4484 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914675    4484 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914680    4484 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914684    4484 flags.go:64] FLAG: --feature-gates=""
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914690    4484 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914695    4484 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914699    4484 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914704    4484 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914709    4484 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914719    4484 flags.go:64] FLAG: --help="false"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914723    4484 flags.go:64] FLAG: --hostname-override="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914732    4484 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914737    4484 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914747    4484 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 18:20:53.921241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914753    4484 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914759    4484 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914764    4484 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914769    4484 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914779    4484 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914783    4484 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914788    4484 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914793    4484 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914797    4484 flags.go:64] FLAG: --kube-reserved=""
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914802    4484 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914806    4484 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914811    4484 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914820    4484 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914824    4484 flags.go:64] FLAG: --lock-file=""
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914829    4484 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914833    4484 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914839    4484 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914850    4484 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914862    4484 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914867    4484 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914905    4484 flags.go:64] FLAG: --logging-format="text"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914929    4484 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914937    4484 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914944    4484 flags.go:64] FLAG: --manifest-url=""
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914949    4484 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 18:20:53.921812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914958    4484 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914964    4484 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914971    4484 flags.go:64] FLAG: --max-pods="110"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.914982    4484 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915009    4484 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915014    4484 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915019    4484 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915026    4484 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915031    4484 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915036    4484 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915443    4484 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915448    4484 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915451    4484 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915587    4484 flags.go:64] FLAG: --pod-cidr=""
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915592    4484 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915595    4484 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915598    4484 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915601    4484 flags.go:64] FLAG: --port="10250"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915604    4484 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915607    4484 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0541ad9bd068e0419"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915610    4484 flags.go:64] FLAG: --qos-reserved=""
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915613    4484 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915616    4484 flags.go:64] FLAG: --register-node="true"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915619    4484 flags.go:64] FLAG: --register-with-taints=""
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915631    4484 flags.go:64] FLAG: --registry-burst="10"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915633    4484 flags.go:64] FLAG: --registry-qps="5"
Oct 31 18:20:53.922410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915637    4484 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915640    4484 flags.go:64] FLAG: --reserved-memory=""
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915645    4484 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915648    4484 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915651    4484 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915653    4484 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915656    4484 flags.go:64] FLAG: --runonce="false"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915659    4484 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915662    4484 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915665    4484 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915668    4484 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915671    4484 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915674    4484 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915678    4484 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915681    4484 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915684    4484 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915686    4484 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915689    4484 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915692    4484 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915695    4484 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915698    4484 flags.go:64] FLAG: --system-cgroups=""
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915701    4484 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915710    4484 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915713    4484 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915716    4484 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 18:20:53.923034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915721    4484 flags.go:64] FLAG: --tls-min-version=""
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915724    4484 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915727    4484 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915730    4484 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915733    4484 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915736    4484 flags.go:64] FLAG: --v="2"
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915741    4484 flags.go:64] FLAG: --version="false"
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915745    4484 flags.go:64] FLAG: --vmodule=""
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915749    4484 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.915754    4484 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915891    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915895    4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915898    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915902    4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915905    4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915908    4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915911    4484 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915914    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915916    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915919    4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915921    4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915923    4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:53.923674 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915926    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915929    4484 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915933    4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915938    4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915941    4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915944    4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915946    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915949    4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915952    4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915955    4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915957    4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915960    4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915962    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915964    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915970    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915972    4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915975    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915977    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915980    4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:53.924183 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915983    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915985    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915988    4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915990    4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915992    4484 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915995    4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.915997    4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916000    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916002    4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916005    4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916007    4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916009    4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916012    4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916014    4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916017    4484 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916019    4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916022    4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916024    4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916026    4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916029    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:53.924657 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916031    4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916033    4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916036    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916038    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916041    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916043    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916046    4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916049    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916052    4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916055    4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916057    4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916060    4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916062    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916065    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916067    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916069    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916072    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916074    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916077    4484 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:53.925197 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916079    4484 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916081    4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916084    4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916086    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916090    4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916093    4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916096    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916098    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916101    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916104    4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916106    4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916108    4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916111    4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916113    4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916116    4484 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916118    4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916120    4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916123    4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916125    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:53.925758 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916127    4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916134    4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916136    4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916139    4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916141    4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916144    4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916146    4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916149    4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916151    4484 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916155    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916157    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916164    4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916167    4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916170    4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916173    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916175    4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:53.926331 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916178    4484 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.916206    4484 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916253    4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916258    4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916261    4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916263    4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916266    4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916268    4484 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916270    4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916273    4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916275    4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916278    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916280    4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916283    4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916286    4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916289    4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:53.927107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916291    4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916295    4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916299    4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916301    4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916303    4484 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916306    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916308    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916311    4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916313    4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916316    4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916320    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916323    4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916325    4484 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916327    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916330    4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916332    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916335    4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916337    4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916339    4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916342    4484 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:53.927694 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916344    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916347    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916349    4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916351    4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916353    4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916356    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916358    4484 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916362    4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916365    4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916367    4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916370    4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916373    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916375    4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916379    4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916385    4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916387    4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916390    4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916392    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916395    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:53.928292 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916398    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916401    4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916404    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916406    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916409    4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916412    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916414    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916417    4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916419    4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916422    4484 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916424    4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916426    4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916429    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916431    4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916433    4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916436    4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916438    4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916440    4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916443    4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:53.928776 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916445    4484 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916447    4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916450    4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916452    4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916454    4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916457    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916459    4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916462    4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916466    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916489    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916494    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916498    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916501    4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916504    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916507    4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916509    4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916512    4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916514    4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916517    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:53.929488 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916520    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916523    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916525    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916527    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916530    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916532    4484 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916535    4484 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916537    4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916539    4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916542    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916546    4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916549    4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916553    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916556    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.916558    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.920223    4484 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.920238    4484 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.920256    4484 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.920261    4484 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 18:20:53.930216 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920315    4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920319    4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920322    4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920325    4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920329    4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920332    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920335    4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920338    4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920340    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920343    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920345    4484 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920348    4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920351    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920353    4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920356    4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920359    4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920361    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920364    4484 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920366    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:53.930770 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920369    4484 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920371    4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920374    4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920377    4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920379    4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920381    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920384    4484 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920387    4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920389    4484 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920392    4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920395    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920398    4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920400    4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920403    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920406    4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920408    4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920411    4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920414    4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920416    4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920419    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:53.931208 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920421    4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920424    4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920426    4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920428    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920431    4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920433    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920436    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920438    4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920440    4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920443    4484 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920445    4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920448    4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920452    4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920456    4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920458    4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920461    4484 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920463    4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920466    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920468    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:53.931851 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920489    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920492    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920494    4484 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920497    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920501    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920504    4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920506    4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920509    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920511    4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920513    4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920516    4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920518    4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920521    4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920524    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920526    4484 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920528    4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920531    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920533    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920535    4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920538    4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:53.932299 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920540    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920542    4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920545    4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920547    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920550    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920552    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920555    4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920557    4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920559    4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920562    4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920564    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920567    4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920569    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920571    4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920574    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920577    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920580    4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920582    4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920585    4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:53.932800 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920587    4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920590    4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920592    4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920594    4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920597    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920599    4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920602    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920604    4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920607    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.920634    4484 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920762    4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920766    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920769    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920772    4484 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920776    4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920778    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:53.933270 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920781    4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920783    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920786    4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920788    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920791    4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920793    4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920795    4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920798    4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920800    4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920803    4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920805    4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920808    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920811    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920813    4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920815    4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920819    4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920822    4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920824    4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920826    4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920829    4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:53.933656 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920831    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920833    4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920836    4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920838    4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920840    4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920844    4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920848    4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920851    4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920853    4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920856    4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920858    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920861    4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920863    4484 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920866    4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920868    4484 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920870    4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920873    4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920875    4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920877    4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920880    4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:53.934122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920882    4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920884    4484 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920886    4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920890    4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920895    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920899    4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920903    4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920906    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920910    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920913    4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920915    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920919    4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920923    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920926    4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920928    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920931    4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920934    4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920936    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920938    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:53.934643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920941    4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920943    4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920945    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920948    4484 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920950    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920953    4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920955    4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920958    4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920960    4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920962    4484 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920964    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920967    4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920969    4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920972    4484 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920974    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920977    4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920979    4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920981    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920984    4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920986    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:53.935081 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920988    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920991    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920994    4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920996    4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.920998    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921001    4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921003    4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921005    4484 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921008    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921010    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921013    4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921015    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921018    4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921020    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921022    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921025    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921027    4484 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921029    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921032    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:53.935555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921034    4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921036    4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.921059    4484 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921096    4484 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921100    4484 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921103    4484 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921105    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921107    4484 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921110    4484 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921112    4484 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921115    4484 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921119    4484 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921123    4484 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921127    4484 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921130    4484 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921133    4484 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 18:20:53.936008 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921136    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921139    4484 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921142    4484 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921144    4484 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921146    4484 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921148    4484 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921151    4484 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921153    4484 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921155    4484 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921158    4484 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921160    4484 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921162    4484 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921165    4484 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921169    4484 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921171    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921174    4484 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921176    4484 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921179    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921181    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921184    4484 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 18:20:53.936390 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921186    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921189    4484 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921193    4484 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921195    4484 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921198    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921200    4484 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921203    4484 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921205    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921207    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921210    4484 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921222    4484 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921225    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921228    4484 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921231    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921233    4484 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921236    4484 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921238    4484 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921241    4484 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921243    4484 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 18:20:53.936863 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921246    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921248    4484 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921251    4484 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921253    4484 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921256    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921259    4484 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921262    4484 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921264    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921267    4484 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921269    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921271    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921274    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921276    4484 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921279    4484 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921281    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921284    4484 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921288    4484 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921290    4484 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921292    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 18:20:53.937297 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921295    4484 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921297    4484 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921300    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921302    4484 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921304    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921307    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921309    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921312    4484 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921315    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921317    4484 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921320    4484 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921322    4484 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921324    4484 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921327    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921330    4484 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921333    4484 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921335    4484 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921338    4484 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921340    4484 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921344    4484 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 18:20:53.937745 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921347    4484 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921349    4484 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921351    4484 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921354    4484 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921356    4484 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921359    4484 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921361    4484 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921363    4484 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921365    4484 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921369    4484 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921371    4484 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921374    4484 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921376    4484 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921379    4484 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:53.921382    4484 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.921430    4484 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.922383    4484 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.922437    4484 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.922640    4484 server.go:1014] "Starting client certificate rotation"
Oct 31 18:20:53.938191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.922778    4484 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 18:20:53.938643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.922834    4484 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 18:15:14 +0000 UTC" deadline="2028-04-30 19:48:18.655322716 +0000 UTC"
Oct 31 18:20:53.938643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.922860    4484 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13129h27m24.732464008s"
Oct 31 18:20:53.938643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.923063    4484 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 18:20:53.938643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.924377    4484 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 18:20:53.938643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.925763    4484 log.go:25] "Validated CRI v1 runtime API"
Oct 31 18:20:53.938643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.927637    4484 log.go:25] "Validated CRI v1 image API"
Oct 31 18:20:53.938643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.928349    4484 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 18:20:53.938643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.930805    4484 fs.go:135] Filesystem UUIDs: map[075e0292-ef41-4f6c-a261-31d551eb0bcb:/dev/nvme0n1p3 21374453-50fe-4038-970c-69c026217ad8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 31 18:20:53.938842 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.930826    4484 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0acb523aa6e0990bcdf23522a8e707265a3204c7cb9c4bad1c4a53145a0ee760/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0acb523aa6e0990bcdf23522a8e707265a3204c7cb9c4bad1c4a53145a0ee760/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fbe0e2b0a44ecb26d14f16c42bccce270e1c85bf6f0499c926351a78c2dcaaa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fbe0e2b0a44ecb26d14f16c42bccce270e1c85bf6f0499c926351a78c2dcaaa/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26745ba601d694844e1c08573e13763f0f2cdcc9e3fa1b5112b8ea65facb680d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26745ba601d694844e1c08573e13763f0f2cdcc9e3fa1b5112b8ea65facb680d/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/364a6458c6c86a01d0cbd92d74698ee653a5b0df3305828cf44eaac37231b16e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/364a6458c6c86a01d0cbd92d74698ee653a5b0df3305828cf44eaac37231b16e/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4db540bbaaa0bc0204d97f5a4aa44acfd0c096e4b2ab77c7acd0db0532f1c614/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4db540bbaaa0bc0204d97f5a4aa44acfd0c096e4b2ab77c7acd0db0532f1c614/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50d556d2fdfe1e392c5679657ad6ba99397cd7bfb673512017eac9b75e0ba4be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50d556d2fdfe1e392c5679657ad6ba99397cd7bfb673512017eac9b75e0ba4be/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c21fc7a7689238d0ddf56d693b8c24f2dfb3b5fc4d85c3c522a28944e2575cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c21fc7a7689238d0ddf56d693b8c24f2dfb3b5fc4d85c3c522a28944e2575cf/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ccdc557353dfd74e57c146fd918fa6b78a570df7ebb22a051ce50dfea83b014/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ccdc557353dfd74e57c146fd918fa6b78a570df7ebb22a051ce50dfea83b014/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88be93709b51a75f67c205065e5a283c303d550473e9cc04242460d77968583b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88be93709b51a75f67c205065e5a283c303d550473e9cc04242460d77968583b/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91b208cc2de4062418d3cebdc99d47d8a42f3fb382db917c2949fd2df9f0c523/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91b208cc2de4062418d3cebdc99d47d8a42f3fb382db917c2949fd2df9f0c523/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/950970b6b956cff6fe7470fa50d663e2dfef38208039af1746ecf60b9094cf9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/950970b6b956cff6fe7470fa50d663e2dfef38208039af1746ecf60b9094cf9d/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d23880c4071a4c19ad010e4d9a9ca1335d7b6e4fe11d4df5dec397b2c02cafe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d23880c4071a4c19ad010e4d9a9ca1335d7b6e4fe11d4df5dec397b2c02cafe/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a330e580cefaa0c74e25f929a39ff7e9c7dfddebfc34f81473e0eea460c3b722/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a330e580cefaa0c74e25f929a39ff7e9c7dfddebfc34f81473e0eea460c3b722/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9817bf28fe482befd8dafed99e9f4250e49297303808ffa69020c612cae69e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9817bf28fe482befd8dafed99e9f4250e49297303808ffa69020c612cae69e9/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dac23bbe9845c5aa0c5a9293b3e821f6f61a65331aab689755505b3f439a455b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dac23bbe9845c5aa0c5a9293b3e821f6f61a65331aab689755505b3f439a455b/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a79a203-4e31-41e5-af54-c8d07320badd/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0a79a203-4e31-41e5-af54-c8d07320badd/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a79a203-4e31-41e5-af54-c8d07320badd/volumes/kubernetes.io~projected/kube-api-access-gg9rk:{mountpoint:/var/lib/kubelet/pods/0a79a203-4e31-41e5-af54-c8d07320badd/volumes/kubernetes.io~projected/kube-api-access-gg9rk major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a79a203-4e31-41e5-af54-c8d07320badd/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0a79a203-4e31-41e5-af54-c8d07320badd/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/203b8b56-b634-4a8b-85b1-46aae63f6dd8/volumes/kubernetes.io~projected/kube-api-access-rhf2p:{mountpoint:/var/lib/kubelet/pods/203b8b56-b634-4a8b-85b1-46aae63f6dd8/volumes/kubernetes.io~projected/kube-api-access-rhf2p major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/207d8f7f-20e9-430d-a74d-42419bc4da86/volumes/kubernetes.io~projected/kube-api-access-5hcq9:{mountpoint:/var/lib/kubelet/pods/207d8f7f-20e9-430d-a74d-42419bc4da86/volumes/kubernetes.io~projected/kube-api-access-5hcq9 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32e9756e-22b8-4adb-a36f-dc837581e01a/volumes/kubernetes.io~projected/kube-api-access-vz9zx:{mountpoint:/var/lib/kubelet/pods/32e9756e-22b8-4adb-a36f-dc837581e01a/volumes/kubernetes.io~projected/kube-api-access-vz9zx major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bfedf5f-370f-425e-88f8-0959db69e660/volumes/kubernetes.io~projected/kube-api-access-xn7jq:{mountpoint:/var/lib/kubelet/pods/3bfedf5f-370f-425e-88f8-0959db69e660/volumes/kubernetes.io~projected/kube-api-access-xn7jq major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/494630d1-d540-42a4-93ab-2b76505246a5/volumes/kubernetes.io~projected/kube-api-access-zl47l:{mountpoint:/var/lib/kubelet/pods/494630d1-d540-42a4-93ab-2b76505246a5/volumes/kubernetes.io~projected/kube-api-access-zl47l major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/828b1043-03ed-45a2-afb3-160da3dce8c0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/828b1043-03ed-45a2-afb3-160da3dce8c0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/828b1043-03ed-45a2-afb3-160da3dce8c0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/828b1043-03ed-45a2-afb3-160da3dce8c0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/828b1043-03ed-45a2-afb3-160da3dce8c0/volumes/kubernetes.io~projected/kube-api-access-g2zvs:{mountpoint:/var/lib/kubelet/pods/828b1043-03ed-45a2-afb3-160da3dce8c0/volumes/kubernetes.io~projected/kube-api-access-g2zvs major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8771b315-47d1-4bd6-b8f7-cc1cb548a789/volumes/kubernetes.io~projected/kube-api-access-hlltr:{mountpoint:/var/lib/kubelet/pods/8771b315-47d1-4bd6-b8f7-cc1cb548a789/volumes/kubernetes.io~projected/kube-api-access-hlltr major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8771b315-47d1-4bd6-b8f7-cc1cb548a789/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8771b315-47d1-4bd6-b8f7-cc1cb548a789/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c12bcbd-bf78-49c1-b60c-0c39e16faf61/volumes/kubernetes.io~projected/kube-api-access-zpzsx:{mountpoint:/var/lib/kubelet/pods/8c12bcbd-bf78-49c1-b60c-0c39e16faf61/volumes/kubernetes.io~projected/kube-api-access-zpzsx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d647747-ac4a-41c4-bec2-f9aff836f5bb/volumes/kubernetes.io~projected/kube-api-access-255ph:{mountpoint:/var/lib/kubelet/pods/9d647747-ac4a-41c4-bec2-f9aff836f5bb/volumes/kubernetes.io~projected/kube-api-access-255ph major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3/volumes/kubernetes.io~projected/kube-api-access-km59v:{mountpoint:/var/lib/kubelet/pods/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3/volumes/kubernetes.io~projected/kube-api-access-km59v major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac38de34-5c48-430d-b06f-7015c64fbb56/volumes/kubernetes.io~projected/kube-api-access-92g26:{mountpoint:/var/lib/kubelet/pods/ac38de34-5c48-430d-b06f-7015c64fbb56/volumes/kubernetes.io~projected/kube-api-access-92g26 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4b27f4e-abf1-407d-a33d-90e23a5fc861/volumes/kubernetes.io~projected/kube-api-access-r68bd:{mountpoint:/var/lib/kubelet/pods/b4b27f4e-abf1-407d-a33d-90e23a5fc861/volumes/kubernetes.io~projected/kube-api-access-r68bd major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0b37043-a08e-475d-b8ff-3c8c2c8b653f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e0b37043-a08e-475d-b8ff-3c8c2c8b653f/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8/volumes/kubernetes.io~projected/kube-api-access-wbdvb:{mountpoint:/var/lib/kubelet/pods/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8/volumes/kubernetes.io~projected/kube-api-access-wbdvb major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8/volumes/kubernetes.io~secret/signing-key major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657/volumes/kubernetes.io~projected/kube-api-access-2v2s4:{mountpoint:/var/lib/kubelet/pods/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657/volumes/kubernetes.io~projected/kube-api-access-2v2s4 major:0 minor:207 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c34afe207a86aa8088fe1d5bca730200a0fef00b41748b676faca351c28f1867/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/745efd40adca3e4d28421202505614df7ac9dab8d33aab7d0104a2335a28ad3c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/cf4beca7a7e08c903fd96ee2b4be890c92eec9905ae04e79e85760ef51205c79/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f49489feb35d407bef33ffaabab45ad9bc8fb7ca1e0e1da6de586ae20247ce07/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a5d7f0023a34058fc431d6c0de3ab9ba4baa33180bdf13dc7dff0bcf559f8a13/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e2eef8b1ad42d387ec4d7836c3a552adc3c0ba5307a2f10c61c4198f8d676806/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bdd1814180eb6850cdcb09881040a0640f62c81fc521f76fb45497b3f25fce18/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8cd15bbc466659349133696d94bce071a6a07229fffc974e4ceb322d7c7a471c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/46b53d3cf3800917f92e993bb578832dc6aed4cb8e8217c12665b268a40fe347/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/2fa7cbc84e097a06585d3d99e6dafd7be9b9d5a6ffbe35fb49072c90477f06a6/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/64da90127deb013c5d8aa082833fced0d2a2c7471cb72f95eec1af40a0fe3c9b/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/2677e18e8491279117bb6c9124192e738dafc20c35a23e13365e0913ff2eb8bf/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/48f77453c08e7f8b2b936fdd1386c2e96cf2c4ea658dfe4bab61fe4d854b788f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/cce70aa01960f225d1107338216bd48e4c8655d7069103a3b97f1f126b746af4/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3a38c4a4c10d699d8c3ec279373dd0db237be998800a4bf85d5f239590cd5cf7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4f99a5c822ded84baa8056c8b753e62e40f5b32e5645805609d7797e0e6f310e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/68b5b4016b9c54dd8f1c16a6ef1098557dd5cffa23f95a0251dde6c8072fa5ef/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/780c8fb1935251f421807c375cb8fcc20116ab49cd9f5e02ba0ff326c6288ff5/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/820cff490ff54338a5636bc75824f7eeb9c99f2d21e8b1f18497efc13a02ce25/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/93aeb38c09bfb6971e036de2ec4cabd7ed1514e9b1ecf5ed701326d0c82100b1/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/fdfdd94ee6dc6e71a7d185b7d3739ad9ab5100a16d6c8d38b7141a18d73271d6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/aed3b0dc7a13341e5c7ebb1783fb5006535a7f8b92511be829ae2082e7536f8b/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/d98e9bf11082d0dca418c5d85dddb407970371b6e8f9f0ad3b8e16d1f5abee1f/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/903067439d618fac8fde602f27a142974630b27c6cdef675ac513992a5990082/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/6cf65a901fb31a077cc4b2bef63477b568de64db42fc6d90ddb8768dc144e708/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/026835ddc99bfd990ceea4641c5dd2f2aeda5793789c6c205bf475e68679db70/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/dad01848b45354ff86a9f2af85bbc449852db0a78f1b5664a9bce0ea5822b1db/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/508880a9e4755bb69a758b02786ea23646f3cedd42218e4664508e5ae4aff8a4/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/afe2e89d9250f67cd1dad8b0c670f12fec75717361aa350a273fd554f8aa67a3/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/1663a84e1a9ab98051719448671fe20d6d08e6725e8f62a7806fc96283883ed5/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/b1968d6cdaac8748e3f5df1edcdc2696987c44d9988339b906ce67189572c0e3/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/15e53c55a8339b5fbdfe77f5d6d809d26715116fec36842ba7a75c63031b44a4/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/da0ce343be693ab8e075bd40f7f25e2592a4d9b045987c5a6d9ebdaed730b93a/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/81d58043bb14e687dfb9dae909865ea566edfc4e69191d4061455b9c7a01a269/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/1bf8c8df356106b07e8afefd53bd0763fcdc8fd9ebc8deaeea35529eeebf3220/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/b8e4e44b1596fb2c99f644444e153599a257a7ba2e247074af820f3d70112d81/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c2f7fd6f97132647b27dba4d4bffe7b8d4576906e704d8199071a02ef83a1926/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/9267668ff68f16d36da5eb4b28c10c4f2bb7eeb3d3dd7fbfc5b4ea05238182d8/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/e6c73dd2909bed7551b3260685dafd4a11046360ecdb91088cd2f5add70d1429/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/9278c86603694d499a06c624bde31e917b8823bbdb6e1ae2cb36dbdfb921c936/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/2bfcad0750c15c32baad7797804f66f8a10c22827d0a1346477dbbd43399f2de/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 18:20:53.938972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.937895    4484 manager.go:217] Machine: {Timestamp:2026-10-31 18:20:53.937442832 +0000 UTC m=+0.083892690 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200011 MemoryCapacity:32812023808 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28740c198459331d15e8d3987a2e2d SystemUUID:ec28740c-1984-5933-1d15-e8d3987a2e2d BootID:b6a2d5a4-2f20-44de-8494-a56a96be2979 Filesystems:[{Device:/var/lib/kubelet/pods/8771b315-47d1-4bd6-b8f7-cc1cb548a789/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/494630d1-d540-42a4-93ab-2b76505246a5/volumes/kubernetes.io~projected/kube-api-access-zl47l DeviceMajor:0 DeviceMinor:54 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26745ba601d694844e1c08573e13763f0f2cdcc9e3fa1b5112b8ea65facb680d/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a79a203-4e31-41e5-af54-c8d07320badd/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9d647747-ac4a-41c4-bec2-f9aff836f5bb/volumes/kubernetes.io~projected/kube-api-access-255ph DeviceMajor:0 DeviceMinor:201 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ccdc557353dfd74e57c146fd918fa6b78a570df7ebb22a051ce50dfea83b014/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/0a79a203-4e31-41e5-af54-c8d07320badd/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a330e580cefaa0c74e25f929a39ff7e9c7dfddebfc34f81473e0eea460c3b722/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8771b315-47d1-4bd6-b8f7-cc1cb548a789/volumes/kubernetes.io~projected/kube-api-access-hlltr DeviceMajor:0 DeviceMinor:197 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3/volumes/kubernetes.io~projected/kube-api-access-km59v DeviceMajor:0 DeviceMinor:206 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/364a6458c6c86a01d0cbd92d74698ee653a5b0df3305828cf44eaac37231b16e/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/8c12bcbd-bf78-49c1-b60c-0c39e16faf61/volumes/kubernetes.io~projected/kube-api-access-zpzsx DeviceMajor:0 DeviceMinor:59 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88be93709b51a75f67c205065e5a283c303d550473e9cc04242460d77968583b/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0acb523aa6e0990bcdf23522a8e707265a3204c7cb9c4bad1c4a53145a0ee760/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657/volumes/kubernetes.io~projected/kube-api-access-2v2s4 DeviceMajor:0 DeviceMinor:207 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406011904 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bfedf5f-370f-425e-88f8-0959db69e660/volumes/kubernetes.io~projected/kube-api-access-xn7jq DeviceMajor:0 DeviceMinor:57 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4db540bbaaa0bc0204d97f5a4aa44acfd0c096e4b2ab77c7acd0db0532f1c614/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4b27f4e-abf1-407d-a33d-90e23a5fc861/volumes/kubernetes.io~projected/kube-api-access-r68bd DeviceMajor:0 DeviceMinor:213 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/203b8b56-b634-4a8b-85b1-46aae63f6dd8/volumes/kubernetes.io~projected/kube-api-access-rhf2p DeviceMajor:0 DeviceMinor:53 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/828b1043-03ed-45a2-afb3-160da3dce8c0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/828b1043-03ed-45a2-afb3-160da3dce8c0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/207d8f7f-20e9-430d-a74d-42419bc4da86/volumes/kubernetes.io~projected/kube-api-access-5hcq9 DeviceMajor:0 DeviceMinor:58 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8/volumes/kubernetes.io~projected/kube-api-access-wbdvb DeviceMajor:0 DeviceMinor:229 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0b37043-a08e-475d-b8ff-3c8c2c8b653f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d23880c4071a4c19ad010e4d9a9ca1335d7b6e4fe11d4df5dec397b2c02cafe/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/950970b6b956cff6fe7470fa50d663e2dfef38208039af1746ecf60b9094cf9d/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:224 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9817bf28fe482befd8dafed99e9f4250e49297303808ffa69020c612cae69e9/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fbe0e2b0a44ecb26d14f16c42bccce270e1c85bf6f0499c926351a78c2dcaaa/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dac23bbe9845c5aa0c5a9293b3e821f6f61a65331aab689755505b3f439a455b/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32e9756e-22b8-4adb-a36f-dc837581e01a/volumes/kubernetes.io~projected/kube-api-access-vz9zx DeviceMajor:0 DeviceMinor:51 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/828b1043-03ed-45a2-afb3-160da3dce8c0/volumes/kubernetes.io~projected/kube-api-access-g2zvs DeviceMajor:0 DeviceMinor:52 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91b208cc2de4062418d3cebdc99d47d8a42f3fb382db917c2949fd2df9f0c523/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c21fc7a7689238d0ddf56d693b8c24f2dfb3b5fc4d85c3c522a28944e2575cf/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:212 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/ac38de34-5c48-430d-b06f-7015c64fbb56/volumes/kubernetes.io~projected/kube-api-access-92g26 DeviceMajor:0 DeviceMinor:55 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50d556d2fdfe1e392c5679657ad6ba99397cd7bfb673512017eac9b75e0ba4be/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/0a79a203-4e31-41e5-af54-c8d07320badd/volumes/kubernetes.io~projected/kube-api-access-gg9rk DeviceMajor:0 DeviceMinor:56 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:364a6458c6c86a0 MacAddress:42:db:50:92:e9:38 Speed:10000 Mtu:8901} {Name:50d556d2fdfe1e3 MacAddress:ee:75:e3:ea:74:f3 Speed:10000 Mtu:8901} {Name:6ccdc557353dfd7 MacAddress:9a:97:cd:e5:c9:14 Speed:10000 Mtu:8901} {Name:91b208cc2de4062 MacAddress:c2:3a:e4:5b:ed:cb Speed:10000 Mtu:8901} {Name:950970b6b956cff MacAddress:9e:bb:9a:74:41:7f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:44:ed:96:a0:3b Speed:0 Mtu:9001} {Name:br-int MacAddress:ba:e5:9e:7a:0d:43 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:44:ed:96:a0:3b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ce:39:de:11:4a:15 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e6:60:11:d6:c2:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812023808 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 18:20:53.938972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.938139    4484 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 18:20:53.938972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.938182    4484 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 18:20:53.938972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.938455    4484 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 18:20:53.938972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.938498    4484 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-120.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 18:20:53.938972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.938630    4484 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 18:20:53.938972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.938637    4484 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 18:20:53.938972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.938650    4484 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 18:20:53.938972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.938668    4484 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 18:20:53.938972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.938748    4484 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 18:20:53.938972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.938908    4484 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 18:20:53.939397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.939022    4484 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 18:20:53.939397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.939054    4484 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 18:20:53.939397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.939064    4484 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 18:20:53.939397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.939074    4484 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 18:20:53.939397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.939083    4484 kubelet.go:405] "Adding apiserver pod source"
Oct 31 18:20:53.939397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.939090    4484 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 18:20:53.939622 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.939610    4484 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 18:20:53.939657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.939622    4484 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 18:20:53.940072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.940051    4484 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 18:20:53.941973 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.941958    4484 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 18:20:53.941973 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.941972    4484 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 18:20:53.942072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942053    4484 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 18:20:53.942246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942235    4484 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 18:20:53.942287 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942263    4484 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 18:20:53.942384 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942374    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 18:20:53.942424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942394    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 18:20:53.942424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942402    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 18:20:53.942424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942408    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 18:20:53.942424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942414    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 18:20:53.942424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942419    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 18:20:53.942424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942424    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 18:20:53.942628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942429    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 18:20:53.942628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942435    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 18:20:53.942628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942441    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 18:20:53.942628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942454    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 18:20:53.942628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942462    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 18:20:53.942628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942508    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 18:20:53.942628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.942514    4484 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 18:20:53.945835 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.945820    4484 server.go:1263] "Started kubelet"
Oct 31 18:20:53.949841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.945935    4484 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 18:20:53.949841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.946178    4484 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 18:20:53.949841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.946589    4484 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 18:20:53.949841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.947347    4484 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 18:20:53.949841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.948188    4484 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 18:20:53.946673 ip-10-0-142-120 systemd[1]: Started Kubernetes Kubelet.
Oct 31 18:20:53.954038 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.954015    4484 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 18:20:53.954128 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.954033    4484 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 18:20:53.954128 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.954108    4484 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 18:15:15 +0000 UTC" deadline="2028-05-03 18:09:36.669515475 +0000 UTC"
Oct 31 18:20:53.954228 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.954126    4484 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13199h48m42.715392308s"
Oct 31 18:20:53.954228 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:53.954161    4484 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-120.us-west-1.compute.internal\" not found"
Oct 31 18:20:53.954228 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.954183    4484 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 18:20:53.954228 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.954198    4484 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 18:20:53.954228 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.954201    4484 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 18:20:53.955610 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955581    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a79a203-4e31-41e5-af54-c8d07320badd" volumeName="kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 18:20:53.955700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955613    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a79a203-4e31-41e5-af54-c8d07320badd" volumeName="kubernetes.io/projected/0a79a203-4e31-41e5-af54-c8d07320badd-kube-api-access-gg9rk" seLinuxMountContext=""
Oct 31 18:20:53.955700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955654    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="203b8b56-b634-4a8b-85b1-46aae63f6dd8" volumeName="kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 18:20:53.955700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955668    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="203b8b56-b634-4a8b-85b1-46aae63f6dd8" volumeName="kubernetes.io/projected/203b8b56-b634-4a8b-85b1-46aae63f6dd8-kube-api-access-rhf2p" seLinuxMountContext=""
Oct 31 18:20:53.955700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955681    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32e9756e-22b8-4adb-a36f-dc837581e01a" volumeName="kubernetes.io/empty-dir/32e9756e-22b8-4adb-a36f-dc837581e01a-tmp-dir" seLinuxMountContext=""
Oct 31 18:20:53.955700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955692    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bfedf5f-370f-425e-88f8-0959db69e660" volumeName="kubernetes.io/projected/3bfedf5f-370f-425e-88f8-0959db69e660-kube-api-access-xn7jq" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955704    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8" volumeName="kubernetes.io/configmap/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-signing-cabundle" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955715    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" volumeName="kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955729    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="494630d1-d540-42a4-93ab-2b76505246a5" volumeName="kubernetes.io/projected/494630d1-d540-42a4-93ab-2b76505246a5-kube-api-access-zl47l" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955740    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" volumeName="kubernetes.io/projected/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-kube-api-access-2v2s4" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955753    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a79a203-4e31-41e5-af54-c8d07320badd" volumeName="kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-env-overrides" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955766    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="494630d1-d540-42a4-93ab-2b76505246a5" volumeName="kubernetes.io/configmap/494630d1-d540-42a4-93ab-2b76505246a5-cni-binary-copy" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955776    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c12bcbd-bf78-49c1-b60c-0c39e16faf61" volumeName="kubernetes.io/projected/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-kube-api-access-zpzsx" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955799    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4c0351a-556c-42f3-b1cc-bdee5bddfaa3" volumeName="kubernetes.io/projected/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-api-access-km59v" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955813    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7" volumeName="kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955825    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0b37043-a08e-475d-b8ff-3c8c2c8b653f" volumeName="kubernetes.io/configmap/e0b37043-a08e-475d-b8ff-3c8c2c8b653f-konnectivity-ca" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955838    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8" volumeName="kubernetes.io/projected/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-kube-api-access-wbdvb" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955858    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" volumeName="kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-service-ca-bundle" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955880    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d647747-ac4a-41c4-bec2-f9aff836f5bb" volumeName="kubernetes.io/projected/9d647747-ac4a-41c4-bec2-f9aff836f5bb-kube-api-access-255ph" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955892    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a79a203-4e31-41e5-af54-c8d07320badd" volumeName="kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-ovnkube-config" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955904    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="207d8f7f-20e9-430d-a74d-42419bc4da86" volumeName="kubernetes.io/configmap/207d8f7f-20e9-430d-a74d-42419bc4da86-iptables-alerter-script" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955916    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="828b1043-03ed-45a2-afb3-160da3dce8c0" volumeName="kubernetes.io/empty-dir/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-tuned" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955929    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8771b315-47d1-4bd6-b8f7-cc1cb548a789" volumeName="kubernetes.io/projected/8771b315-47d1-4bd6-b8f7-cc1cb548a789-kube-api-access-hlltr" seLinuxMountContext=""
Oct 31 18:20:53.955941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955940    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac38de34-5c48-430d-b06f-7015c64fbb56" volumeName="kubernetes.io/projected/ac38de34-5c48-430d-b06f-7015c64fbb56-kube-api-access-92g26" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955951    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0b37043-a08e-475d-b8ff-3c8c2c8b653f" volumeName="kubernetes.io/secret/e0b37043-a08e-475d-b8ff-3c8c2c8b653f-agent-certs" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955962    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" volumeName="kubernetes.io/empty-dir/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-archives-path" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955974    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d647747-ac4a-41c4-bec2-f9aff836f5bb" volumeName="kubernetes.io/configmap/9d647747-ac4a-41c4-bec2-f9aff836f5bb-config-volume" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.955987    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d647747-ac4a-41c4-bec2-f9aff836f5bb" volumeName="kubernetes.io/empty-dir/9d647747-ac4a-41c4-bec2-f9aff836f5bb-tmp-dir" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956007    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4c0351a-556c-42f3-b1cc-bdee5bddfaa3" volumeName="kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956021    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="828b1043-03ed-45a2-afb3-160da3dce8c0" volumeName="kubernetes.io/empty-dir/828b1043-03ed-45a2-afb3-160da3dce8c0-tmp" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956031    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4c0351a-556c-42f3-b1cc-bdee5bddfaa3" volumeName="kubernetes.io/empty-dir/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-data-volume" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956042    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4b27f4e-abf1-407d-a33d-90e23a5fc861" volumeName="kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956054    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="203b8b56-b634-4a8b-85b1-46aae63f6dd8" volumeName="kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956065    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="494630d1-d540-42a4-93ab-2b76505246a5" volumeName="kubernetes.io/configmap/494630d1-d540-42a4-93ab-2b76505246a5-multus-daemon-config" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956083    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="828b1043-03ed-45a2-afb3-160da3dce8c0" volumeName="kubernetes.io/projected/828b1043-03ed-45a2-afb3-160da3dce8c0-kube-api-access-g2zvs" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956094    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8771b315-47d1-4bd6-b8f7-cc1cb548a789" volumeName="kubernetes.io/secret/8771b315-47d1-4bd6-b8f7-cc1cb548a789-cert" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956106    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a79a203-4e31-41e5-af54-c8d07320badd" volumeName="kubernetes.io/secret/0a79a203-4e31-41e5-af54-c8d07320badd-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956118    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="203b8b56-b634-4a8b-85b1-46aae63f6dd8" volumeName="kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cni-binary-copy" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956130    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="207d8f7f-20e9-430d-a74d-42419bc4da86" volumeName="kubernetes.io/projected/207d8f7f-20e9-430d-a74d-42419bc4da86-kube-api-access-5hcq9" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956141    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32e9756e-22b8-4adb-a36f-dc837581e01a" volumeName="kubernetes.io/projected/32e9756e-22b8-4adb-a36f-dc837581e01a-kube-api-access-vz9zx" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956153    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bfedf5f-370f-425e-88f8-0959db69e660" volumeName="kubernetes.io/configmap/3bfedf5f-370f-425e-88f8-0959db69e660-serviceca" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956165    4484 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8" volumeName="kubernetes.io/secret/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-signing-key" seLinuxMountContext=""
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956176    4484 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956184    4484 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956222    4484 factory.go:55] Registering systemd factory
Oct 31 18:20:53.956700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.956235    4484 factory.go:223] Registration of the systemd container factory successfully
Oct 31 18:20:53.957439 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.957185    4484 factory.go:153] Registering CRI-O factory
Oct 31 18:20:53.957439 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.957202    4484 factory.go:223] Registration of the crio container factory successfully
Oct 31 18:20:53.957439 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.957263    4484 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 18:20:53.957439 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.957285    4484 factory.go:103] Registering Raw factory
Oct 31 18:20:53.957439 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.957300    4484 manager.go:1199] Started watching for new ooms in manager
Oct 31 18:20:53.957710 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.957689    4484 manager.go:319] Starting recovery of all containers
Oct 31 18:20:53.973870 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.973839    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:20:53.974748 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.974722    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:20:53.984355 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.984324    4484 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 18:20:53.985652 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.985632    4484 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 18:20:53.985744 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.985660    4484 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 18:20:53.985744 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.985686    4484 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 18:20:53.985848 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:53.985732    4484 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 18:20:53.998247 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:53.998221    4484 generic.go:356] "Generic (PLEG): container finished" podID="26fabc6b6751b80acc2b46148e9c9b23" containerID="f9a639036e9a74c230169c2ab53bf7fd10574f0a67faf4e736e343bf8388da23" exitCode=0
Oct 31 18:20:54.017597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.017571    4484 generic.go:356] "Generic (PLEG): container finished" podID="203b8b56-b634-4a8b-85b1-46aae63f6dd8" containerID="6f0a8d9755332d7eff87318f35f238797cfe9f4c8289d7c5e0401d584fbff4a4" exitCode=0
Oct 31 18:20:54.017597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.017595    4484 generic.go:356] "Generic (PLEG): container finished" podID="203b8b56-b634-4a8b-85b1-46aae63f6dd8" containerID="0e3ad3389f416863c1d8dc6dde1c328b9c8818f1ec73c95c75e2cc62fbb40982" exitCode=0
Oct 31 18:20:54.017597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.017603    4484 generic.go:356] "Generic (PLEG): container finished" podID="203b8b56-b634-4a8b-85b1-46aae63f6dd8" containerID="456e16070e2e6f7b2cc0471ab07e2750eb3dbdf1ce7abb5f2231691ad9e6cf20" exitCode=0
Oct 31 18:20:54.017797 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.017608    4484 generic.go:356] "Generic (PLEG): container finished" podID="203b8b56-b634-4a8b-85b1-46aae63f6dd8" containerID="8fd79eff7a36c2af19b51d5da1207ad190fe174f88e5939e1b4934a82f5a8d9c" exitCode=0
Oct 31 18:20:54.017797 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.017613    4484 generic.go:356] "Generic (PLEG): container finished" podID="203b8b56-b634-4a8b-85b1-46aae63f6dd8" containerID="db257033c0c4f9aeab92cb24613a828e0657b9625b9ae5eef6ca99f573a608eb" exitCode=0
Oct 31 18:20:54.017797 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.017618    4484 generic.go:356] "Generic (PLEG): container finished" podID="203b8b56-b634-4a8b-85b1-46aae63f6dd8" containerID="6d977e44663439ed83b97b0307158a35d2a1918cab7f323dd64b876d551e31a6" exitCode=0
Oct 31 18:20:54.024510 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.024490    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:20:54.024858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.024840    4484 generic.go:356] "Generic (PLEG): container finished" podID="0a79a203-4e31-41e5-af54-c8d07320badd" containerID="7c18a3c4daf52914848e65ff05ea03367b5189175fc8e34ebc92757192cf978e" exitCode=1
Oct 31 18:20:54.025877 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.025862    4484 manager.go:324] Recovery completed
Oct 31 18:20:54.038727 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.038678    4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:54.039905 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.039866    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:54.039982 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.039928    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:54.039982 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.039965    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:54.042403 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042386    4484 cpu_manager.go:226] "Starting" policy="none"
Oct 31 18:20:54.042403 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042401    4484 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 18:20:54.042532 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042406    4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:54.042532 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042419    4484 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 18:20:54.042618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042588    4484 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 18:20:54.042618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042600    4484 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 18:20:54.042618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042616    4484 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 18:20:54.042712 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042621    4484 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 18:20:54.042712 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042627    4484 policy_none.go:50] "Start"
Oct 31 18:20:54.042712 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042635    4484 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 18:20:54.042712 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042645    4484 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 18:20:54.042834 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042760    4484 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 18:20:54.042834 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042767    4484 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 18:20:54.042834 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.042772    4484 policy_none.go:44] "Start"
Oct 31 18:20:54.043208 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.043194    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:54.043260 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.043222    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:54.043260 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.043232    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:54.047225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.047195    4484 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 18:20:54.050731 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.050714    4484 manager.go:341] "Starting Device Plugin manager"
Oct 31 18:20:54.050825 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:54.050745    4484 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 18:20:54.050825 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.050757    4484 server.go:87] "Starting device plugin registration server"
Oct 31 18:20:54.051047 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.051035    4484 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 18:20:54.051078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.051048    4484 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 18:20:54.051181 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.051160    4484 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 18:20:54.051308 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.051268    4484 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 18:20:54.051417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.051313    4484 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 18:20:54.051417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.051320    4484 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 18:20:54.051527 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.051379    4484 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T18:20:54.051302429Z","UUID":"7ecc3001-1473-47dd-9715-7e2628293522","Handler":null,"Name":"","Endpoint":""}
Oct 31 18:20:54.052349 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:54.052331    4484 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 18:20:54.052441 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:54.052390    4484 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-120.us-west-1.compute.internal\" not found"
Oct 31 18:20:54.052760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.052745    4484 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 18:20:54.052844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.052776    4484 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 18:20:54.086978 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.086528    4484 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"]
Oct 31 18:20:54.087093 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.087078    4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:54.087929 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.087914    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:54.088003 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.087941    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:54.088003 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.087950    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:54.088175 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.088165    4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:54.088329 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.088317    4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:54.088985 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.088972    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:54.088985 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.088979    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:54.089071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.088994    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:54.089071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.089002    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:54.089071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.089013    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:54.089071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.089004    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:54.089250 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.089209    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal" event={"ID":"26fabc6b6751b80acc2b46148e9c9b23","Type":"ContainerStarted","Data":"af952c6d5252ab870afd96a35b3a3cc910db8c5b6618dd1bb387b451b7b48e33"}
Oct 31 18:20:54.089299 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:54.089252    4484 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-120.us-west-1.compute.internal\" not found" node="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.089299 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.089257    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal" event={"ID":"26fabc6b6751b80acc2b46148e9c9b23","Type":"ContainerDied","Data":"f9a639036e9a74c230169c2ab53bf7fd10574f0a67faf4e736e343bf8388da23"}
Oct 31 18:20:54.089363 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.089309    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal" event={"ID":"26fabc6b6751b80acc2b46148e9c9b23","Type":"ContainerStarted","Data":"88be93709b51a75f67c205065e5a283c303d550473e9cc04242460d77968583b"}
Oct 31 18:20:54.089363 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.089319    4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:54.089424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.089386    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal" event={"ID":"eb7aca0bcf277ba7408a15c0b72c2378","Type":"ContainerStarted","Data":"5bf28ecd9bca61815d391c14f9bcfc3bb453f30431715b85d9d0c87987503fd9"}
Oct 31 18:20:54.089424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.089409    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal" event={"ID":"eb7aca0bcf277ba7408a15c0b72c2378","Type":"ContainerStarted","Data":"d9817bf28fe482befd8dafed99e9f4250e49297303808ffa69020c612cae69e9"}
Oct 31 18:20:54.090099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.090074    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:54.090165 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.090103    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:54.090165 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.090117    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:54.090270 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:54.090260    4484 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-120.us-west-1.compute.internal\" not found" node="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.148311 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.148278    4484 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:54.148311 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.148293    4484 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:54.148513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.148453    4484 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 18:20:54.150167 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.150155    4484 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:54.151229 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.151217    4484 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 18:20:54.152195 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.152172    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 18:20:54.152195 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.152195    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 18:20:54.152321 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.152209    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 18:20:54.152321 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.152257    4484 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.154673 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.154642    4484 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.201032 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.201012    4484 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 18:20:54.249371 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.249348    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26fabc6b6751b80acc2b46148e9c9b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"26fabc6b6751b80acc2b46148e9c9b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.249501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.249380    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fabc6b6751b80acc2b46148e9c9b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"26fabc6b6751b80acc2b46148e9c9b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.249501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.249429    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb7aca0bcf277ba7408a15c0b72c2378-config\") pod \"kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"eb7aca0bcf277ba7408a15c0b72c2378\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.266980 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.266959    4484 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 18:20:54.267106 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:54.266999    4484 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.267106 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.267025    4484 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.315870 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.315808    4484 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.315989 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.315879    4484 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.316447 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.316428    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeNotReady"
Oct 31 18:20:54.316584 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.316465    4484 setters.go:546] "Node became not ready" node="ip-10-0-142-120.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T18:20:54Z","lastTransitionTime":"2026-10-31T18:20:54Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 31 18:20:54.316584 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.316571    4484 nodeinfomanager.go:471] existing CSINode "ip-10-0-142-120.us-west-1.compute.internal" is owned by different node (oldNodeID="625973a8-7444-48d2-81b8-be98e2a2d793", newNodeID=""), cleaning up...
Oct 31 18:20:54.350139 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.350112    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26fabc6b6751b80acc2b46148e9c9b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"26fabc6b6751b80acc2b46148e9c9b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.350257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.350141    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fabc6b6751b80acc2b46148e9c9b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"26fabc6b6751b80acc2b46148e9c9b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.350257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.350159    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb7aca0bcf277ba7408a15c0b72c2378-config\") pod \"kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"eb7aca0bcf277ba7408a15c0b72c2378\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.350257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.350196    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb7aca0bcf277ba7408a15c0b72c2378-config\") pod \"kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"eb7aca0bcf277ba7408a15c0b72c2378\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.350257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.350224    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26fabc6b6751b80acc2b46148e9c9b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"26fabc6b6751b80acc2b46148e9c9b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.350257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.350239    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fabc6b6751b80acc2b46148e9c9b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" (UID: \"26fabc6b6751b80acc2b46148e9c9b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.369265 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.369237    4484 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 18:20:54.369385 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:54.369316    4484 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:54.587225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.587150    4484 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-120.us-west-1.compute.internal" event="NodeReady"
Oct 31 18:20:54.587384 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.587241    4484 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 18:20:54.940278 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:54.940210    4484 apiserver.go:52] "Watching apiserver"
Oct 31 18:20:55.028008 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.027986    4484 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:55.028134 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.028099    4484 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:55.095061 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.095038    4484 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 18:20:55.095290 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.095273    4484 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 18:20:55.095387 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.095303    4484 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 18:20:55.095387 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:55.095322    4484 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:55.095387 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:55.095336    4484 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal"
Oct 31 18:20:55.095558 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.095540    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-xpvmh","openshift-dns/node-resolver-tjxmm","openshift-image-registry/node-ca-g8ng5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-120.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-rjntd","kube-system/konnectivity-agent-82t6b","kube-system/kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal","openshift-multus/multus-nlk9v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6","openshift-ovn-kubernetes/ovnkube-node-vsrqn","openshift-ingress-canary/ingress-canary-wqsdj","openshift-insights/insights-runtime-extractor-gwg8k","kube-system/global-pull-secret-syncer-h72m5","openshift-insights/periodic-gathering-vlnmf-hb4xw","openshift-multus/network-metrics-daemon-r92q2","openshift-cluster-node-tuning-operator/tuned-gpjxn","openshift-network-diagnostics/network-check-target-9mlcf","openshift-service-ca/service-ca-5fffc97bdf-qxdz6","openshift-dns/dns-default-glvbh"]
Oct 31 18:20:55.098175 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.098153    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:55.102707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.102655    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:55.103341 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.103307    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.150749 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.150726    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 18:20:55.151077 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.151057    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 18:20:55.151118 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.151104    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 18:20:55.151118 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.151104    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 18:20:55.151206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.151147    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-27pxv\""
Oct 31 18:20:55.151206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.151152    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 18:20:55.151206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.151185    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 18:20:55.151381 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.151371    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgxxf\""
Oct 31 18:20:55.152703 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.152690    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 18:20:55.152988 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.152977    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-szhfk\""
Oct 31 18:20:55.155184 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.155165    4484 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 18:20:55.188216 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.188192    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:55.250120 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.250096    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 18:20:55.250258 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.250122    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 18:20:55.250308 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.250258    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 18:20:55.250427 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.250414    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 18:20:55.250501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.250429    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 18:20:55.250663 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.250649    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 18:20:55.250737 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.250671    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 18:20:55.250737 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.250719    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 18:20:55.250854 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.250829    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 18:20:55.250957 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.250873    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 18:20:55.250957 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.250880    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 18:20:55.251185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251174    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 18:20:55.251238 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251225    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 18:20:55.251269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251234    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 18:20:55.251269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251255    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nkjx2\""
Oct 31 18:20:55.251333 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251281    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 18:20:55.251333 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251319    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 18:20:55.251333 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251321    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 18:20:55.251418 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251336    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 18:20:55.251418 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251329    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 18:20:55.251418 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251345    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 18:20:55.251541 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251448    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 18:20:55.251541 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251488    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 18:20:55.251676 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251663    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 18:20:55.251740 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251728    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 18:20:55.251789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251760    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 18:20:55.251789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251775    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 18:20:55.251789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251779    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 18:20:55.251889 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251809    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 18:20:55.251935 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251901    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 18:20:55.251935 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251911    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9xfhz\""
Oct 31 18:20:55.252026 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251956    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 18:20:55.252026 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251960    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 18:20:55.252026 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251979    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l4kdr\""
Oct 31 18:20:55.252026 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251988    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 18:20:55.252026 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251996    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 18:20:55.252026 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.251992    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 18:20:55.252212 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.252156    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m5pkq\""
Oct 31 18:20:55.255414 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255397    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.255488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255421    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-ovn\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.255488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255436    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cnibin\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.255488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255453    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.255679 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255556    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v2s4\" (UniqueName: \"kubernetes.io/projected/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-kube-api-access-2v2s4\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:55.255679 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255581    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-crio-socket\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.255679 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255601    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0b37043-a08e-475d-b8ff-3c8c2c8b653f-agent-certs\") pod \"konnectivity-agent-82t6b\" (UID: \"e0b37043-a08e-475d-b8ff-3c8c2c8b653f\") " pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:55.255679 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255627    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-cni-multus\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.255679 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255646    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-cni-bin\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.255679 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255663    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-env-overrides\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.255679 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255678    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92g26\" (UniqueName: \"kubernetes.io/projected/ac38de34-5c48-430d-b06f-7015c64fbb56-kube-api-access-92g26\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:55.255954 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255709    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-sys-fs\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.255954 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255789    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-etc-kubernetes\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.255954 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255827    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a79a203-4e31-41e5-af54-c8d07320badd-ovn-node-metrics-cert\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.255954 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255899    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-env-overrides\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.255954 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255945    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bfedf5f-370f-425e-88f8-0959db69e660-serviceca\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:55.256125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255969    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-var-lib-kubelet\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.256125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.255992    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zvs\" (UniqueName: \"kubernetes.io/projected/828b1043-03ed-45a2-afb3-160da3dce8c0-kube-api-access-g2zvs\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.256125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256005    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a79a203-4e31-41e5-af54-c8d07320badd-ovn-node-metrics-cert\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.256125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256100    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhf2p\" (UniqueName: \"kubernetes.io/projected/203b8b56-b634-4a8b-85b1-46aae63f6dd8-kube-api-access-rhf2p\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.256280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256148    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-signing-cabundle\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:55.256280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256175    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/207d8f7f-20e9-430d-a74d-42419bc4da86-host-slash\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:55.256280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256198    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-tuned\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.256280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256227    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-registration-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.256280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256229    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3bfedf5f-370f-425e-88f8-0959db69e660-serviceca\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:55.256280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256250    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 18:20:55.256280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256271    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.256572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256295    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-kubelet\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.256572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256320    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5t2xq\""
Oct 31 18:20:55.256572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256347    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:55.256572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256377    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7jq\" (UniqueName: \"kubernetes.io/projected/3bfedf5f-370f-425e-88f8-0959db69e660-kube-api-access-xn7jq\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:55.256572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256400    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/207d8f7f-20e9-430d-a74d-42419bc4da86-iptables-alerter-script\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:55.256572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256421    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-sys\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.256572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256431    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-signing-cabundle\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:55.256572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256425    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-tuned\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.256572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256447    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-lib-modules\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.256821 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256672    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6wm76\""
Oct 31 18:20:55.256821 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256770    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8771b315-47d1-4bd6-b8f7-cc1cb548a789-cert\") pod \"ingress-canary-wqsdj\" (UID: \"8771b315-47d1-4bd6-b8f7-cc1cb548a789\") " pod="openshift-ingress-canary/ingress-canary-wqsdj"
Oct 31 18:20:55.256821 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256793    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/494630d1-d540-42a4-93ab-2b76505246a5-cni-binary-copy\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.256821 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256809    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-cni-bin\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.256947 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256824    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-var-lib-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.256947 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256844    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-service-ca-bundle\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:55.256947 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256883    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg9rk\" (UniqueName: \"kubernetes.io/projected/0a79a203-4e31-41e5-af54-c8d07320badd-kube-api-access-gg9rk\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.256947 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256900    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/207d8f7f-20e9-430d-a74d-42419bc4da86-iptables-alerter-script\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:55.257156 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.256950    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hcq9\" (UniqueName: \"kubernetes.io/projected/207d8f7f-20e9-430d-a74d-42419bc4da86-kube-api-access-5hcq9\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:55.257156 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257040    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8771b315-47d1-4bd6-b8f7-cc1cb548a789-cert\") pod \"ingress-canary-wqsdj\" (UID: \"8771b315-47d1-4bd6-b8f7-cc1cb548a789\") " pod="openshift-ingress-canary/ingress-canary-wqsdj"
Oct 31 18:20:55.257156 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257058    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-trusted-ca-bundle\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:55.257156 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257076    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-service-ca-bundle\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:55.257292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257156    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.257292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257190    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km59v\" (UniqueName: \"kubernetes.io/projected/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-api-access-km59v\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.257292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257192    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/494630d1-d540-42a4-93ab-2b76505246a5-cni-binary-copy\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.257292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257225    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-kubelet\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.257292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257253    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cni-binary-copy\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.257292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257278    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-kubelet-config\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:55.257549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257299    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbdvb\" (UniqueName: \"kubernetes.io/projected/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-kube-api-access-wbdvb\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:55.257549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257342    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.257549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257343    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-data-volume\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.257549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257414    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-data-volume\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.257549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257412    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysconfig\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.257549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257498    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-netns\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.257549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257517    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cni-binary-copy\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.257549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257528    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-dbus\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:55.257822 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257565    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:55.257822 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257605    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-host\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.257822 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257639    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpzsx\" (UniqueName: \"kubernetes.io/projected/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-kube-api-access-zpzsx\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.257822 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257676    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.257822 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257754    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-ovnkube-config\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.257822 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257789    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-system-cni-dir\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.258018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257838    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32e9756e-22b8-4adb-a36f-dc837581e01a-hosts-file\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:55.258018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257857    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz9zx\" (UniqueName: \"kubernetes.io/projected/32e9756e-22b8-4adb-a36f-dc837581e01a-kube-api-access-vz9zx\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:55.258018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257871    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-kubernetes\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.258018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257922    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-systemd\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.258018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.257975    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-ovnkube-config\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.258018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258000    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.258269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258028    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-multus-certs\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.258269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258051    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl47l\" (UniqueName: \"kubernetes.io/projected/494630d1-d540-42a4-93ab-2b76505246a5-kube-api-access-zl47l\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.258269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258104    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-node-log\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.258269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258203    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-modprobe-d\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.258269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258230    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-run\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.258449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258279    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-socket-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.258449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258304    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-cni-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.258449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258331    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-conf-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.258449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258347    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ms9mk\""
Oct 31 18:20:55.258449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258366    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-run-netns\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.258449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258393    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-etc-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.258449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258400    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ghldp\""
Oct 31 18:20:55.258449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258421    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.258449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258446    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68bd\" (UniqueName: \"kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd\") pod \"network-check-target-9mlcf\" (UID: \"b4b27f4e-abf1-407d-a33d-90e23a5fc861\") " pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258462    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-signing-key\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258540    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-hostroot\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258563    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-log-socket\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258589    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d647747-ac4a-41c4-bec2-f9aff836f5bb-config-volume\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258631    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d647747-ac4a-41c4-bec2-f9aff836f5bb-tmp-dir\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258633    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258668    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysctl-conf\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258711    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d647747-ac4a-41c4-bec2-f9aff836f5bb-tmp-dir\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258722    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/828b1043-03ed-45a2-afb3-160da3dce8c0-tmp\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258747    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-signing-key\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258747    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-system-cni-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258762    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/828b1043-03ed-45a2-afb3-160da3dce8c0-tmp\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258787    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-socket-dir-parent\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258803    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-systemd\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258820    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255ph\" (UniqueName: \"kubernetes.io/projected/9d647747-ac4a-41c4-bec2-f9aff836f5bb-kube-api-access-255ph\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258802    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d647747-ac4a-41c4-bec2-f9aff836f5bb-config-volume\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:55.258859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258849    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258871    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-archives-path\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258887    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258903    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0b37043-a08e-475d-b8ff-3c8c2c8b653f-konnectivity-ca\") pod \"konnectivity-agent-82t6b\" (UID: \"e0b37043-a08e-475d-b8ff-3c8c2c8b653f\") " pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258930    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlltr\" (UniqueName: \"kubernetes.io/projected/8771b315-47d1-4bd6-b8f7-cc1cb548a789-kube-api-access-hlltr\") pod \"ingress-canary-wqsdj\" (UID: \"8771b315-47d1-4bd6-b8f7-cc1cb548a789\") " pod="openshift-ingress-canary/ingress-canary-wqsdj"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258960    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-run-ovn-kubernetes\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.258975    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-archives-path\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259021    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-cni-netd\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259047    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-ovnkube-script-lib\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259069    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bfedf5f-370f-425e-88f8-0959db69e660-host\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259091    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-device-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259119    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-cnibin\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259155    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-systemd-units\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259156    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0b37043-a08e-475d-b8ff-3c8c2c8b653f-konnectivity-ca\") pod \"konnectivity-agent-82t6b\" (UID: \"e0b37043-a08e-475d-b8ff-3c8c2c8b653f\") " pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259182    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-slash\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259208    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.259296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259239    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e9756e-22b8-4adb-a36f-dc837581e01a-tmp-dir\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:55.259717 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259339    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/203b8b56-b634-4a8b-85b1-46aae63f6dd8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.259717 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259346    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysctl-d\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.259717 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259371    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e9756e-22b8-4adb-a36f-dc837581e01a-tmp-dir\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:55.259717 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259401    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-k8s-cni-cncf-io\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.259717 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259418    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/494630d1-d540-42a4-93ab-2b76505246a5-multus-daemon-config\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.259717 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.259567    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/494630d1-d540-42a4-93ab-2b76505246a5-multus-daemon-config\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.261378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.261364    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vv64h\""
Oct 31 18:20:55.300803 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.300785    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xr757\""
Oct 31 18:20:55.300803 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.300792    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-77xvx\""
Oct 31 18:20:55.300893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.300842    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7mm2n\""
Oct 31 18:20:55.300893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.300857    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w82k4\""
Oct 31 18:20:55.300893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.300874    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 18:20:55.300893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.300878    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 18:20:55.300893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.300876    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-72cf5\""
Oct 31 18:20:55.301116 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.301101    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 18:20:55.302715 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.302700    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 18:20:55.306791 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.306777    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0b37043-a08e-475d-b8ff-3c8c2c8b653f-agent-certs\") pod \"konnectivity-agent-82t6b\" (UID: \"e0b37043-a08e-475d-b8ff-3c8c2c8b653f\") " pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:20:55.306838 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.306812    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-original-pull-secret\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:55.308400 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.308383    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-trusted-ca-bundle\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:55.310123 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.310105    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a79a203-4e31-41e5-af54-c8d07320badd-ovnkube-script-lib\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.312593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.312572    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhf2p\" (UniqueName: \"kubernetes.io/projected/203b8b56-b634-4a8b-85b1-46aae63f6dd8-kube-api-access-rhf2p\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.312647 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.312615    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92g26\" (UniqueName: \"kubernetes.io/projected/ac38de34-5c48-430d-b06f-7015c64fbb56-kube-api-access-92g26\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:55.312854 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.312841    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zvs\" (UniqueName: \"kubernetes.io/projected/828b1043-03ed-45a2-afb3-160da3dce8c0-kube-api-access-g2zvs\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.313387 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.313373    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7jq\" (UniqueName: \"kubernetes.io/projected/3bfedf5f-370f-425e-88f8-0959db69e660-kube-api-access-xn7jq\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:55.313419 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.313389    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v2s4\" (UniqueName: \"kubernetes.io/projected/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-kube-api-access-2v2s4\") pod \"periodic-gathering-vlnmf-hb4xw\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") " pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:20:55.313510 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.313497    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg9rk\" (UniqueName: \"kubernetes.io/projected/0a79a203-4e31-41e5-af54-c8d07320badd-kube-api-access-gg9rk\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.316675 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.316622    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255ph\" (UniqueName: \"kubernetes.io/projected/9d647747-ac4a-41c4-bec2-f9aff836f5bb-kube-api-access-255ph\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:55.316780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.316677    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl47l\" (UniqueName: \"kubernetes.io/projected/494630d1-d540-42a4-93ab-2b76505246a5-kube-api-access-zl47l\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.316780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.316775    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hcq9\" (UniqueName: \"kubernetes.io/projected/207d8f7f-20e9-430d-a74d-42419bc4da86-kube-api-access-5hcq9\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:55.316879 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.316841    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km59v\" (UniqueName: \"kubernetes.io/projected/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-kube-api-access-km59v\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.316926 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.316886    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpzsx\" (UniqueName: \"kubernetes.io/projected/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-kube-api-access-zpzsx\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.317086 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.317066    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlltr\" (UniqueName: \"kubernetes.io/projected/8771b315-47d1-4bd6-b8f7-cc1cb548a789-kube-api-access-hlltr\") pod \"ingress-canary-wqsdj\" (UID: \"8771b315-47d1-4bd6-b8f7-cc1cb548a789\") " pod="openshift-ingress-canary/ingress-canary-wqsdj"
Oct 31 18:20:55.317386 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.317071    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbdvb\" (UniqueName: \"kubernetes.io/projected/f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8-kube-api-access-wbdvb\") pod \"service-ca-5fffc97bdf-qxdz6\" (UID: \"f0fb4572-e09f-4de5-b9ba-b7bbd4a306c8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qxdz6"
Oct 31 18:20:55.317386 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.317326    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz9zx\" (UniqueName: \"kubernetes.io/projected/32e9756e-22b8-4adb-a36f-dc837581e01a-kube-api-access-vz9zx\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:55.319238 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.319221    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68bd\" (UniqueName: \"kubernetes.io/projected/b4b27f4e-abf1-407d-a33d-90e23a5fc861-kube-api-access-r68bd\") pod \"network-check-target-9mlcf\" (UID: \"b4b27f4e-abf1-407d-a33d-90e23a5fc861\") " pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:55.321853 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.321836    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wqsdj_8771b315-47d1-4bd6-b8f7-cc1cb548a789/serve-healthcheck-canary/0.log"
Oct 31 18:20:55.359688 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.359666    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/207d8f7f-20e9-430d-a74d-42419bc4da86-host-slash\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:55.359776 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.359764    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/207d8f7f-20e9-430d-a74d-42419bc4da86-host-slash\") pod \"iptables-alerter-rjntd\" (UID: \"207d8f7f-20e9-430d-a74d-42419bc4da86\") " pod="openshift-network-operator/iptables-alerter-rjntd"
Oct 31 18:20:55.359858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.359840    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-registration-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.360088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.359693    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-registration-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.360127 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360101    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.360127 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360117    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-kubelet\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.360190 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360131    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-sys\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.360190 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360145    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-lib-modules\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.360190 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360159    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-cni-bin\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.360190 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360173    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-var-lib-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.360190 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360185    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-kubelet\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360197    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-kubelet\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360211    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-etc-selinux\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360218    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-kubelet-config\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360216    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-cni-bin\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360240    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-var-lib-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360253    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-sys\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360257    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-kubelet\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360263    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-kubelet-config\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360281    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysconfig\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360289    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-lib-modules\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360303    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-netns\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360319    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-dbus\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360333    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360348    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-host\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360353    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysconfig\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360369    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360371    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7-dbus\") pod \"global-pull-secret-syncer-h72m5\" (UID: \"c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7\") " pod="kube-system/global-pull-secret-syncer-h72m5"
Oct 31 18:20:55.360382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360369    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-netns\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360401    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360418    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-system-cni-dir\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360419    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-host\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360434    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32e9756e-22b8-4adb-a36f-dc837581e01a-hosts-file\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360454    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-kubernetes\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360467    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-system-cni-dir\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360489    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-systemd\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360514    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32e9756e-22b8-4adb-a36f-dc837581e01a-hosts-file\") pod \"node-resolver-tjxmm\" (UID: \"32e9756e-22b8-4adb-a36f-dc837581e01a\") " pod="openshift-dns/node-resolver-tjxmm"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360532    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-systemd\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360547    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360556    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-kubernetes\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360574    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-multus-certs\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360587    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360597    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-node-log\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360620    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-modprobe-d\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360625    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-multus-certs\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360642    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-run\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360660    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-node-log\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360665    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-socket-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360669    4484 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360701    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-run\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360726    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-modprobe-d\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360748    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-cni-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360750    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-socket-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360692    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-cni-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360836    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-conf-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360854    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-run-netns\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360870    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-etc-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360888    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-hostroot\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360909    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-log-socket\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360929    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-run-netns\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360934    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysctl-conf\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360957    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-conf-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360961    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-system-cni-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360986    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-etc-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360993    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-system-cni-dir\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360995    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-socket-dir-parent\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360996    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-log-socket\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.360969    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-hostroot\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361037    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysctl-conf\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361043    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-systemd\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361056    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-multus-socket-dir-parent\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361069    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:55.361088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361098    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-systemd\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361193    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361212    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-run-ovn-kubernetes\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361241    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-cni-netd\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361263    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bfedf5f-370f-425e-88f8-0959db69e660-host\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361282    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-device-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:55.361289    4484 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:55.361299    4484 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361306    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-run-ovn-kubernetes\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361312    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-cni-netd\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361346    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-cnibin\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361354    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bfedf5f-370f-425e-88f8-0959db69e660-host\") pod \"node-ca-g8ng5\" (UID: \"3bfedf5f-370f-425e-88f8-0959db69e660\") " pod="openshift-image-registry/node-ca-g8ng5"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361302    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-cnibin\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:55.361391    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:55.86136863 +0000 UTC m=+2.007818489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361392    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-device-dir\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:55.361439    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs podName:ac38de34-5c48-430d-b06f-7015c64fbb56 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:55.861426286 +0000 UTC m=+2.007876133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs") pod "network-metrics-daemon-r92q2" (UID: "ac38de34-5c48-430d-b06f-7015c64fbb56") : secret "metrics-daemon-secret" not found
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361453    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-systemd-units\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361468    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-slash\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361506    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysctl-d\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361506    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-systemd-units\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361521    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-k8s-cni-cncf-io\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361537    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361550    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-ovn\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361562    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-slash\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361580    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cnibin\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361598    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-openvswitch\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361602    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361605    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-run-k8s-cni-cncf-io\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361608    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-run-ovn\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361616    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-etc-sysctl-d\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361636    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-crio-socket\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361648    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-cnibin\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361653    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/203b8b56-b634-4a8b-85b1-46aae63f6dd8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xpvmh\" (UID: \"203b8b56-b634-4a8b-85b1-46aae63f6dd8\") " pod="openshift-multus/multus-additional-cni-plugins-xpvmh"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361665    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-cni-multus\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361722    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-crio-socket\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361793    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-cni-bin\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.361889    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a79a203-4e31-41e5-af54-c8d07320badd-host-cni-bin\") pod \"ovnkube-node-vsrqn\" (UID: \"0a79a203-4e31-41e5-af54-c8d07320badd\") " pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.362073    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-sys-fs\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.362110    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-etc-kubernetes\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.362140    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-var-lib-kubelet\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.362202    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-etc-kubernetes\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.362820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.362256    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/828b1043-03ed-45a2-afb3-160da3dce8c0-var-lib-kubelet\") pod \"tuned-gpjxn\" (UID: \"828b1043-03ed-45a2-afb3-160da3dce8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-gpjxn"
Oct 31 18:20:55.365685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.364499    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/494630d1-d540-42a4-93ab-2b76505246a5-host-var-lib-cni-multus\") pod \"multus-nlk9v\" (UID: \"494630d1-d540-42a4-93ab-2b76505246a5\") " pod="openshift-multus/multus-nlk9v"
Oct 31 18:20:55.365685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.365585    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d647747-ac4a-41c4-bec2-f9aff836f5bb-metrics-tls\") pod \"dns-default-glvbh\" (UID: \"9d647747-ac4a-41c4-bec2-f9aff836f5bb\") " pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:55.365816 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.365758    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c12bcbd-bf78-49c1-b60c-0c39e16faf61-sys-fs\") pod \"aws-ebs-csi-driver-node-vjfg6\" (UID: \"8c12bcbd-bf78-49c1-b60c-0c39e16faf61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vjfg6"
Oct 31 18:20:55.403430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.403412    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:55.790832 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.790763    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-glvbh"]
Oct 31 18:20:55.833428 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:20:55.833391    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d647747_ac4a_41c4_bec2_f9aff836f5bb.slice/crio-b83595f4b55f73a3433345962f905e93266f28a87b0b1d2f1c19974db25a3ad2 WatchSource:0}: Error finding container b83595f4b55f73a3433345962f905e93266f28a87b0b1d2f1c19974db25a3ad2: Status 404 returned error can't find the container with id b83595f4b55f73a3433345962f905e93266f28a87b0b1d2f1c19974db25a3ad2
Oct 31 18:20:55.835636 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.835616    4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:20:55.866282 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.866252    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:55.866371 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:55.866287    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:55.866404 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:55.866396    4484 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:55.866446 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:55.866399    4484 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 18:20:55.866499 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:55.866446    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:56.866433557 +0000 UTC m=+3.012883399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:55.866499 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:55.866459    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs podName:ac38de34-5c48-430d-b06f-7015c64fbb56 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:56.866453103 +0000 UTC m=+3.012902945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs") pod "network-metrics-daemon-r92q2" (UID: "ac38de34-5c48-430d-b06f-7015c64fbb56") : secret "metrics-daemon-secret" not found
Oct 31 18:20:56.031357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:56.031320    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-glvbh" event={"ID":"9d647747-ac4a-41c4-bec2-f9aff836f5bb","Type":"ContainerStarted","Data":"b83595f4b55f73a3433345962f905e93266f28a87b0b1d2f1c19974db25a3ad2"}
Oct 31 18:20:56.182487 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:56.182405    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:56.398610 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:56.398573    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:56.556536 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:56.556497    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:56.724673 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:56.724638    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:56.874282 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:56.874206    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:56.874282 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:56.874243    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:56.874489 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:56.874348    4484 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:56.874489 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:56.874355    4484 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 18:20:56.874489 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:56.874408    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:58.874391133 +0000 UTC m=+5.020840980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:56.874489 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:56.874422    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs podName:ac38de34-5c48-430d-b06f-7015c64fbb56 nodeName:}" failed. No retries permitted until 2026-10-31 18:20:58.874415596 +0000 UTC m=+5.020865439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs") pod "network-metrics-daemon-r92q2" (UID: "ac38de34-5c48-430d-b06f-7015c64fbb56") : secret "metrics-daemon-secret" not found
Oct 31 18:20:56.884102 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:56.884068    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:57.046096 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:57.046058    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:58.022219 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:58.022184    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:58.039771 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:58.039740    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-glvbh" event={"ID":"9d647747-ac4a-41c4-bec2-f9aff836f5bb","Type":"ContainerStarted","Data":"425f28c1fffa37d29b65e52c24ba031dab10e8d49c88120d652ba2683ca79b4e"}
Oct 31 18:20:58.039886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:58.039796    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-glvbh" event={"ID":"9d647747-ac4a-41c4-bec2-f9aff836f5bb","Type":"ContainerStarted","Data":"d95f76f1c239314d9c22ee81e5c33c476d7c74b0757deadc2284d3d42036e2eb"}
Oct 31 18:20:58.039921 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:58.039884    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-glvbh"
Oct 31 18:20:58.229945 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:58.229868    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:58.373813 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:58.373775    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:58.575740 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:58.575696    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:58.795219 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:58.795182    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:58.891394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:58.891321    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:20:58.891394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:58.891355    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:20:58.891587 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:58.891452    4484 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:58.891587 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:58.891455    4484 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 18:20:58.891587 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:58.891527    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls podName:a4c0351a-556c-42f3-b1cc-bdee5bddfaa3 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:02.891515262 +0000 UTC m=+9.037965104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwg8k" (UID: "a4c0351a-556c-42f3-b1cc-bdee5bddfaa3") : secret "insights-runtime-extractor-tls" not found
Oct 31 18:20:58.891587 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:20:58.891540    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs podName:ac38de34-5c48-430d-b06f-7015c64fbb56 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:02.891534112 +0000 UTC m=+9.037983954 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs") pod "network-metrics-daemon-r92q2" (UID: "ac38de34-5c48-430d-b06f-7015c64fbb56") : secret "metrics-daemon-secret" not found
Oct 31 18:20:58.983862 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:58.983832    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:59.589431 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:59.589390    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:59.639661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:59.639631    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:20:59.781103 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:59.781072    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:20:59.975574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:20:59.975509    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:21:00.178937 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:00.178899    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:00.395821 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:00.395790    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:00.584326 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:00.584283    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:00.789808 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:00.789774    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:00.984368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:00.984340    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:01.648644 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:01.648617    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:21:01.648828 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:01.648817    4484 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 18:21:02.185386 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:02.185355    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:02.388847 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:02.388811    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:02.583271 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:02.583229    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:02.779997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:02.779945    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:02.920258 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:02.920170    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:21:02.920258 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:02.920209    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:21:02.920527 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:21:02.920325    4484 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 18:21:02.920527 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:21:02.920418    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs podName:ac38de34-5c48-430d-b06f-7015c64fbb56 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:10.9203972 +0000 UTC m=+17.066847046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs") pod "network-metrics-daemon-r92q2" (UID: "ac38de34-5c48-430d-b06f-7015c64fbb56") : secret "metrics-daemon-secret" not found
Oct 31 18:21:02.922668 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:02.922647    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a4c0351a-556c-42f3-b1cc-bdee5bddfaa3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwg8k\" (UID: \"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3\") " pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:21:02.984826 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:02.984791    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:03.203857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:03.203784    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gwg8k"
Oct 31 18:21:03.291394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:03.290259    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vsrqn"
Oct 31 18:21:03.676833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:03.676808    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gwg8k"]
Oct 31 18:21:03.679240 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:21:03.679213    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c0351a_556c_42f3_b1cc_bdee5bddfaa3.slice/crio-3840e04cd8e477eb16420975bb09bfb101d7ee1a471f93a885dd9622a9cf8c84 WatchSource:0}: Error finding container 3840e04cd8e477eb16420975bb09bfb101d7ee1a471f93a885dd9622a9cf8c84: Status 404 returned error can't find the container with id 3840e04cd8e477eb16420975bb09bfb101d7ee1a471f93a885dd9622a9cf8c84
Oct 31 18:21:04.058852 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:04.058817    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwg8k" event={"ID":"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3","Type":"ContainerStarted","Data":"3c35c4e97d18986174f96d098aa69faafcf0e89df99a22defa2e202904d8013a"}
Oct 31 18:21:04.059016 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:04.058871    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwg8k" event={"ID":"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3","Type":"ContainerStarted","Data":"3840e04cd8e477eb16420975bb09bfb101d7ee1a471f93a885dd9622a9cf8c84"}
Oct 31 18:21:04.190006 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:04.189962    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:04.396381 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:04.396308    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:04.549843 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:04.549808    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:04.715275 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:04.715200    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:04.884900 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:04.884872    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:05.055074 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:05.055045    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:05.063488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:05.063443    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwg8k" event={"ID":"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3","Type":"ContainerStarted","Data":"c57fe0b23ef1c806e1d9ddb77f3b4c68ef9d719398f71ac44afaaf244c082be9"}
Oct 31 18:21:07.072101 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:07.072068    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwg8k" event={"ID":"a4c0351a-556c-42f3-b1cc-bdee5bddfaa3","Type":"ContainerStarted","Data":"5e5abab8274b374fd3c0d3419ff0d8a4556284e813a62809a9ed2c1496247ab4"}
Oct 31 18:21:08.044194 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:08.044163    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-glvbh"
Oct 31 18:21:10.255407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.255367    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf"]
Oct 31 18:21:10.259445 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.259425    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf"
Oct 31 18:21:10.314658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.314630    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 18:21:10.315203 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.315186    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gl4z9\""
Oct 31 18:21:10.365536 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.365511    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 18:21:10.366221 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.366206    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf"]
Oct 31 18:21:10.366578 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.366563    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bfc6c5d55-h9bc5"]
Oct 31 18:21:10.368158 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.368140    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.370376 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.370355    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kc5x\" (UniqueName: \"kubernetes.io/projected/5b721371-4513-40ca-88e7-d35b9be0f942-kube-api-access-5kc5x\") pod \"migrator-66df5d496b-wtdbf\" (UID: \"5b721371-4513-40ca-88e7-d35b9be0f942\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf"
Oct 31 18:21:10.449493 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.449445    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bfc6c5d55-h9bc5"]
Oct 31 18:21:10.471412 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.471391    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44e537b7-7adb-411c-925b-ea3b9de73f65-bound-sa-token\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.471576 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.471426    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44e537b7-7adb-411c-925b-ea3b9de73f65-ca-trust-extracted\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.471576 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.471459    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44e537b7-7adb-411c-925b-ea3b9de73f65-registry-tls\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.471687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.471602    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44e537b7-7adb-411c-925b-ea3b9de73f65-installation-pull-secrets\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.471687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.471649    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kc5x\" (UniqueName: \"kubernetes.io/projected/5b721371-4513-40ca-88e7-d35b9be0f942-kube-api-access-5kc5x\") pod \"migrator-66df5d496b-wtdbf\" (UID: \"5b721371-4513-40ca-88e7-d35b9be0f942\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf"
Oct 31 18:21:10.471687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.471678    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44e537b7-7adb-411c-925b-ea3b9de73f65-registry-certificates\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.471811 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.471730    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44e537b7-7adb-411c-925b-ea3b9de73f65-image-registry-private-configuration\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.471811 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.471783    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44e537b7-7adb-411c-925b-ea3b9de73f65-trusted-ca\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.471910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.471823    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxglj\" (UniqueName: \"kubernetes.io/projected/44e537b7-7adb-411c-925b-ea3b9de73f65-kube-api-access-bxglj\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.539063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.539004    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 18:21:10.550101 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.550070    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sqcvs\""
Oct 31 18:21:10.550206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.550140    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 18:21:10.550206 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:21:10.550151    4484 projected.go:291] Couldn't get configMap openshift-kube-storage-version-migrator/openshift-service-ca.crt: configmap "openshift-service-ca.crt" not found
Oct 31 18:21:10.550206 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:21:10.550168    4484 projected.go:196] Error preparing data for projected volume kube-api-access-5kc5x for pod openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf: configmap "openshift-service-ca.crt" not found
Oct 31 18:21:10.550206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.550184    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 18:21:10.550386 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:21:10.550226    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b721371-4513-40ca-88e7-d35b9be0f942-kube-api-access-5kc5x podName:5b721371-4513-40ca-88e7-d35b9be0f942 nodeName:}" failed. No retries permitted until 2026-10-31 18:21:11.05021072 +0000 UTC m=+17.196660574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5kc5x" (UniqueName: "kubernetes.io/projected/5b721371-4513-40ca-88e7-d35b9be0f942-kube-api-access-5kc5x") pod "migrator-66df5d496b-wtdbf" (UID: "5b721371-4513-40ca-88e7-d35b9be0f942") : configmap "openshift-service-ca.crt" not found
Oct 31 18:21:10.550386 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.550141    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 18:21:10.557152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.557128    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:10.573139 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.573112    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44e537b7-7adb-411c-925b-ea3b9de73f65-installation-pull-secrets\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.573225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.573155    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44e537b7-7adb-411c-925b-ea3b9de73f65-registry-certificates\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.573225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.573198    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44e537b7-7adb-411c-925b-ea3b9de73f65-image-registry-private-configuration\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.573225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.573220    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44e537b7-7adb-411c-925b-ea3b9de73f65-trusted-ca\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.573362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.573250    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxglj\" (UniqueName: \"kubernetes.io/projected/44e537b7-7adb-411c-925b-ea3b9de73f65-kube-api-access-bxglj\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.573362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.573284    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44e537b7-7adb-411c-925b-ea3b9de73f65-bound-sa-token\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.573362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.573308    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44e537b7-7adb-411c-925b-ea3b9de73f65-ca-trust-extracted\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.573362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.573354    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44e537b7-7adb-411c-925b-ea3b9de73f65-registry-tls\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.573922 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.573896    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/44e537b7-7adb-411c-925b-ea3b9de73f65-ca-trust-extracted\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.574364 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.574337    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/44e537b7-7adb-411c-925b-ea3b9de73f65-trusted-ca\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.574449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.574375    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/44e537b7-7adb-411c-925b-ea3b9de73f65-registry-certificates\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.575829 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.575806    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/44e537b7-7adb-411c-925b-ea3b9de73f65-installation-pull-secrets\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.575829 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.575823    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/44e537b7-7adb-411c-925b-ea3b9de73f65-registry-tls\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.575938 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.575843    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/44e537b7-7adb-411c-925b-ea3b9de73f65-image-registry-private-configuration\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.634206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.634181    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxglj\" (UniqueName: \"kubernetes.io/projected/44e537b7-7adb-411c-925b-ea3b9de73f65-kube-api-access-bxglj\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.634407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.634390    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44e537b7-7adb-411c-925b-ea3b9de73f65-bound-sa-token\") pod \"image-registry-bfc6c5d55-h9bc5\" (UID: \"44e537b7-7adb-411c-925b-ea3b9de73f65\") " pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.677600 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.677571    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:10.789046 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.789009    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:10.976803 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.976728    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:21:10.979291 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.979275    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac38de34-5c48-430d-b06f-7015c64fbb56-metrics-certs\") pod \"network-metrics-daemon-r92q2\" (UID: \"ac38de34-5c48-430d-b06f-7015c64fbb56\") " pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:21:10.980757 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.980729    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:10.998707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:10.998681    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r92q2"
Oct 31 18:21:11.077199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:11.077168    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kc5x\" (UniqueName: \"kubernetes.io/projected/5b721371-4513-40ca-88e7-d35b9be0f942-kube-api-access-5kc5x\") pod \"migrator-66df5d496b-wtdbf\" (UID: \"5b721371-4513-40ca-88e7-d35b9be0f942\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf"
Oct 31 18:21:11.079397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:11.079373    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kc5x\" (UniqueName: \"kubernetes.io/projected/5b721371-4513-40ca-88e7-d35b9be0f942-kube-api-access-5kc5x\") pod \"migrator-66df5d496b-wtdbf\" (UID: \"5b721371-4513-40ca-88e7-d35b9be0f942\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf"
Oct 31 18:21:11.095838 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:11.095821    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bfc6c5d55-h9bc5"]
Oct 31 18:21:11.098034 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:21:11.098012    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e537b7_7adb_411c_925b_ea3b9de73f65.slice/crio-c20eb9a75c067c0dbea2114256d0113fc97399e0f2fca9589bf69e3d15b0ed4f WatchSource:0}: Error finding container c20eb9a75c067c0dbea2114256d0113fc97399e0f2fca9589bf69e3d15b0ed4f: Status 404 returned error can't find the container with id c20eb9a75c067c0dbea2114256d0113fc97399e0f2fca9589bf69e3d15b0ed4f
Oct 31 18:21:11.147921 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:11.147897    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:11.169174 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:11.169155    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf"
Oct 31 18:21:11.362977 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:11.362948    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r92q2"]
Oct 31 18:21:11.366974 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:21:11.366945    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac38de34_5c48_430d_b06f_7015c64fbb56.slice/crio-b729e2a225a1b0d115142d7c313cc59dd81b33090627f87d197d27b32ee491fe WatchSource:0}: Error finding container b729e2a225a1b0d115142d7c313cc59dd81b33090627f87d197d27b32ee491fe: Status 404 returned error can't find the container with id b729e2a225a1b0d115142d7c313cc59dd81b33090627f87d197d27b32ee491fe
Oct 31 18:21:11.533845 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:11.533803    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf"]
Oct 31 18:21:11.536371 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:21:11.536347    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b721371_4513_40ca_88e7_d35b9be0f942.slice/crio-5df442e5a385d0fe38a42447daa3248388fa014ebaddda6c357972025d529d2c WatchSource:0}: Error finding container 5df442e5a385d0fe38a42447daa3248388fa014ebaddda6c357972025d529d2c: Status 404 returned error can't find the container with id 5df442e5a385d0fe38a42447daa3248388fa014ebaddda6c357972025d529d2c
Oct 31 18:21:12.090586 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:12.090551    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r92q2" event={"ID":"ac38de34-5c48-430d-b06f-7015c64fbb56","Type":"ContainerStarted","Data":"b729e2a225a1b0d115142d7c313cc59dd81b33090627f87d197d27b32ee491fe"}
Oct 31 18:21:12.092057 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:12.092033    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5" event={"ID":"44e537b7-7adb-411c-925b-ea3b9de73f65","Type":"ContainerStarted","Data":"283281642ba3e72d21ffb4bb1bc52d18b59aab30a958d6b4606b86767dfc0c84"}
Oct 31 18:21:12.092162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:12.092071    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5" event={"ID":"44e537b7-7adb-411c-925b-ea3b9de73f65","Type":"ContainerStarted","Data":"c20eb9a75c067c0dbea2114256d0113fc97399e0f2fca9589bf69e3d15b0ed4f"}
Oct 31 18:21:12.092162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:12.092117    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:12.093043 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:12.093027    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf" event={"ID":"5b721371-4513-40ca-88e7-d35b9be0f942","Type":"ContainerStarted","Data":"5df442e5a385d0fe38a42447daa3248388fa014ebaddda6c357972025d529d2c"}
Oct 31 18:21:12.211728 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:12.211679    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5" podStartSLOduration=2.21166723 podStartE2EDuration="2.21166723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:12.211441998 +0000 UTC m=+18.357891861" watchObservedRunningTime="2026-10-31 18:21:12.21166723 +0000 UTC m=+18.358117087"
Oct 31 18:21:13.098857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:13.098817    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r92q2" event={"ID":"ac38de34-5c48-430d-b06f-7015c64fbb56","Type":"ContainerStarted","Data":"281abbec5f5f3253fc9ea192039d4c05f107d65dbbed936bb7b3ad3583992d17"}
Oct 31 18:21:13.099299 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:13.098883    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r92q2" event={"ID":"ac38de34-5c48-430d-b06f-7015c64fbb56","Type":"ContainerStarted","Data":"f8caeba656176f55e0dad315223db4d70465d72399af81e6d3db152f88c30025"}
Oct 31 18:21:14.102806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:14.102766    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf" event={"ID":"5b721371-4513-40ca-88e7-d35b9be0f942","Type":"ContainerStarted","Data":"f3a2eaa18051c1a7fe7866fb5ca867676c591bc5fb2782139eb629e1b899a778"}
Oct 31 18:21:14.159912 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:14.159879    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:14.217185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:14.217141    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wtdbf" podStartSLOduration=2.589992862 podStartE2EDuration="4.21712981s" totalImagesPullingTime="1.627136948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:14.216989428 +0000 UTC m=+20.363439295" watchObservedRunningTime="2026-10-31 18:21:14.21712981 +0000 UTC m=+20.363579673"
Oct 31 18:21:14.323588 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:14.323561    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:14.492633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:14.492595    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:14.647219 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:14.647190    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:15.469752 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:15.469715    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:15.624882 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:15.624849    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:15.780789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:15.780756    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:16.514385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:16.514348    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:16.574950 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:16.574917    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c"]
Oct 31 18:21:16.577189 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:16.577173    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c"
Oct 31 18:21:16.616066 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:16.616037    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d9298c7e-36d1-4ee0-862e-d3d875b8c1f6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bww7c\" (UID: \"d9298c7e-36d1-4ee0-862e-d3d875b8c1f6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c"
Oct 31 18:21:16.664665 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:16.664637    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9dm42\""
Oct 31 18:21:16.680506 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:16.680467    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 18:21:16.681353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:16.681329    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c"]
Oct 31 18:21:16.715924 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:16.715851    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:16.716704 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:16.716689    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d9298c7e-36d1-4ee0-862e-d3d875b8c1f6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bww7c\" (UID: \"d9298c7e-36d1-4ee0-862e-d3d875b8c1f6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c"
Oct 31 18:21:16.719114 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:16.719088    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d9298c7e-36d1-4ee0-862e-d3d875b8c1f6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bww7c\" (UID: \"d9298c7e-36d1-4ee0-862e-d3d875b8c1f6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c"
Oct 31 18:21:16.886385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:16.886346    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c"
Oct 31 18:21:17.251102 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:17.251074    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c"]
Oct 31 18:21:17.254122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:21:17.254098    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9298c7e_36d1_4ee0_862e_d3d875b8c1f6.slice/crio-cc81841e2e423802a6558b7d1d53ce82c6eec3cda88e1df00f4ff36afddd538c WatchSource:0}: Error finding container cc81841e2e423802a6558b7d1d53ce82c6eec3cda88e1df00f4ff36afddd538c: Status 404 returned error can't find the container with id cc81841e2e423802a6558b7d1d53ce82c6eec3cda88e1df00f4ff36afddd538c
Oct 31 18:21:18.116233 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:18.116196    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c" event={"ID":"d9298c7e-36d1-4ee0-862e-d3d875b8c1f6","Type":"ContainerStarted","Data":"cc81841e2e423802a6558b7d1d53ce82c6eec3cda88e1df00f4ff36afddd538c"}
Oct 31 18:21:18.758519 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:18.758467    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:18.907740 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:18.907708    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:19.061605 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:19.061528    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:19.752078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:19.752054    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-82t6b"
Oct 31 18:21:19.846352 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:19.846318    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:20.085656 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:20.085624    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:20.124425 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:20.124397    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c" event={"ID":"d9298c7e-36d1-4ee0-862e-d3d875b8c1f6","Type":"ContainerStarted","Data":"0eb50c407cb86ae379ee8c90cd4860e82ba85badd5a275f71b67ffc63d1876d2"}
Oct 31 18:21:20.124588 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:20.124455    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c"
Oct 31 18:21:20.129216 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:20.129198    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c"
Oct 31 18:21:20.242060 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:20.242014    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bww7c" podStartSLOduration=1.8181969919999998 podStartE2EDuration="4.242001804s" totalImagesPullingTime="2.423804812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:20.24151359 +0000 UTC m=+26.387963452" watchObservedRunningTime="2026-10-31 18:21:20.242001804 +0000 UTC m=+26.388451667"
Oct 31 18:21:20.276188 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:20.276160    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:20.444921 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:20.444844    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:20.650238 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:20.650206    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:20.815411 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:20.815383    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:21.014168 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:21.014140    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:21.208198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:21.208120    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:21.388084 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:21.388054    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:21.555723 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:21.555693    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:21.759141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:21.759107    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:21.909990 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:21.909900    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:22.854448 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:22.854408    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:23.062615 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:23.062581    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:24.978257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:24.978222    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:25.188215 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:25.188183    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:25.466891 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:21:25.466852    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:21:26.049085 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:26.049048    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:26.211506 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:26.211453    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:26.757142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:26.757111    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:27.156257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:27.156178    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:27.546706 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:27.546675    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:27.718037 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:27.718007    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:27.898173 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:27.898100    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:27.955122 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:27.955088    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f8cn5"]
Oct 31 18:21:27.981956 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:27.981934    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.049159 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.049137    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 18:21:28.071016 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.070989    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:28.085649 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.085621    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 18:21:28.086145 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.086049    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9l6d9\""
Oct 31 18:21:28.089710 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.089690    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-wtmp\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.089818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.089735    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-tls\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.089818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.089759    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91cf1206-44e3-4b7b-84a7-f6be0821bab4-metrics-client-ca\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.089902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.089857    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91cf1206-44e3-4b7b-84a7-f6be0821bab4-root\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.089981 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.089967    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.090015 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.089991    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77ch4\" (UniqueName: \"kubernetes.io/projected/91cf1206-44e3-4b7b-84a7-f6be0821bab4-kube-api-access-77ch4\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.090058 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.090047    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-accelerators-collector-config\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.090093 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.090066    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91cf1206-44e3-4b7b-84a7-f6be0821bab4-sys\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.090093 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.090084    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-textfile\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.118599 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.118579    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 18:21:28.136344 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.136325    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 18:21:28.136344 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.136335    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 18:21:28.136344 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.136335    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 18:21:28.190632 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.190570    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-tls\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.190632 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.190600    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91cf1206-44e3-4b7b-84a7-f6be0821bab4-metrics-client-ca\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.190632 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.190618    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91cf1206-44e3-4b7b-84a7-f6be0821bab4-root\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.191007 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.190676    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91cf1206-44e3-4b7b-84a7-f6be0821bab4-root\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.191007 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.190819    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.191007 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.190851    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77ch4\" (UniqueName: \"kubernetes.io/projected/91cf1206-44e3-4b7b-84a7-f6be0821bab4-kube-api-access-77ch4\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.191007 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.190879    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-accelerators-collector-config\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.191007 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.190902    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91cf1206-44e3-4b7b-84a7-f6be0821bab4-sys\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.191007 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.190925    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-textfile\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.191007 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.190961    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-wtmp\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.191007 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.190990    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91cf1206-44e3-4b7b-84a7-f6be0821bab4-sys\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.191383 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.191115    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-wtmp\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.191383 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.191275    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91cf1206-44e3-4b7b-84a7-f6be0821bab4-metrics-client-ca\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.191383 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.191312    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-textfile\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.191651 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.191631    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-accelerators-collector-config\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.193014 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.192990    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-tls\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.193102 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.193058    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91cf1206-44e3-4b7b-84a7-f6be0821bab4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.251165 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.251140    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77ch4\" (UniqueName: \"kubernetes.io/projected/91cf1206-44e3-4b7b-84a7-f6be0821bab4-kube-api-access-77ch4\") pod \"node-exporter-f8cn5\" (UID: \"91cf1206-44e3-4b7b-84a7-f6be0821bab4\") " pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.266792 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.266757    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:28.332643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.332617    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f8cn5"
Oct 31 18:21:28.375529 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:21:28.375499    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cf1206_44e3_4b7b_84a7_f6be0821bab4.slice/crio-23dc0f835d2d82b4fd547e3df7d8cd98bc33117ac53f760557f51b893acc4aee WatchSource:0}: Error finding container 23dc0f835d2d82b4fd547e3df7d8cd98bc33117ac53f760557f51b893acc4aee: Status 404 returned error can't find the container with id 23dc0f835d2d82b4fd547e3df7d8cd98bc33117ac53f760557f51b893acc4aee
Oct 31 18:21:28.438778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:28.438742    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:29.150593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:29.150562    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f8cn5" event={"ID":"91cf1206-44e3-4b7b-84a7-f6be0821bab4","Type":"ContainerStarted","Data":"23dc0f835d2d82b4fd547e3df7d8cd98bc33117ac53f760557f51b893acc4aee"}
Oct 31 18:21:29.272217 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:29.272179    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:29.475786 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:29.475707    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:29.642609 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:29.642585    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9mlcf"
Oct 31 18:21:29.652849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:29.652813    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:30.132368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:30.132335    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:31.158036 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:31.158002    4484 generic.go:356] "Generic (PLEG): container finished" podID="91cf1206-44e3-4b7b-84a7-f6be0821bab4" containerID="952a26d4f0eb29c1a4343a7db9dbc66b6697be162b3fb1afc40ca9714aca3243" exitCode=0
Oct 31 18:21:31.158432 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:31.158066    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f8cn5" event={"ID":"91cf1206-44e3-4b7b-84a7-f6be0821bab4","Type":"ContainerDied","Data":"952a26d4f0eb29c1a4343a7db9dbc66b6697be162b3fb1afc40ca9714aca3243"}
Oct 31 18:21:32.011131 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.011093    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:32.163442 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.163408    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f8cn5" event={"ID":"91cf1206-44e3-4b7b-84a7-f6be0821bab4","Type":"ContainerStarted","Data":"d18e0f15bc38911a02273d631c12b10c067f365aca0d53b65e096606b45bfc86"}
Oct 31 18:21:32.163859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.163492    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f8cn5" event={"ID":"91cf1206-44e3-4b7b-84a7-f6be0821bab4","Type":"ContainerStarted","Data":"d339542e6c8d6434056e6f36832d00d35e22b6c02a154072cc38a5182a2a5387"}
Oct 31 18:21:32.186886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.186856    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:32.288807 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.288722    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f8cn5" podStartSLOduration=3.140835723 podStartE2EDuration="5.288710617s" totalImagesPullingTime="2.065572673s" totalInitContainerRuntime="82.302221ms" isStatefulPod=true podCreationTimestamp="2026-10-31 18:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:32.288252326 +0000 UTC m=+38.434702190" watchObservedRunningTime="2026-10-31 18:21:32.288710617 +0000 UTC m=+38.435160459"
Oct 31 18:21:32.352362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.352322    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:32.522518 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.522485    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:32.600819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.600751    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"]
Oct 31 18:21:32.602701 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.602685    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"
Oct 31 18:21:32.656134 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.656103    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dhvl7\""
Oct 31 18:21:32.656262 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.656177    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 18:21:32.661223 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.661203    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"]
Oct 31 18:21:32.688230 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.688199    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:32.724261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.724235    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls7v6\" (UniqueName: \"kubernetes.io/projected/56dbaf32-7147-4622-afef-37fb8375bd4f-kube-api-access-ls7v6\") pod \"monitoring-plugin-77f4f76594-nj5kx\" (UID: \"56dbaf32-7147-4622-afef-37fb8375bd4f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"
Oct 31 18:21:32.724361 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.724266    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/56dbaf32-7147-4622-afef-37fb8375bd4f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nj5kx\" (UID: \"56dbaf32-7147-4622-afef-37fb8375bd4f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"
Oct 31 18:21:32.825516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.825490    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls7v6\" (UniqueName: \"kubernetes.io/projected/56dbaf32-7147-4622-afef-37fb8375bd4f-kube-api-access-ls7v6\") pod \"monitoring-plugin-77f4f76594-nj5kx\" (UID: \"56dbaf32-7147-4622-afef-37fb8375bd4f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"
Oct 31 18:21:32.825636 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.825523    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/56dbaf32-7147-4622-afef-37fb8375bd4f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nj5kx\" (UID: \"56dbaf32-7147-4622-afef-37fb8375bd4f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"
Oct 31 18:21:32.828218 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.828194    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/56dbaf32-7147-4622-afef-37fb8375bd4f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nj5kx\" (UID: \"56dbaf32-7147-4622-afef-37fb8375bd4f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"
Oct 31 18:21:32.869631 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.869569    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:32.883210 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.883191    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls7v6\" (UniqueName: \"kubernetes.io/projected/56dbaf32-7147-4622-afef-37fb8375bd4f-kube-api-access-ls7v6\") pod \"monitoring-plugin-77f4f76594-nj5kx\" (UID: \"56dbaf32-7147-4622-afef-37fb8375bd4f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"
Oct 31 18:21:32.912156 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:32.912136    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"
Oct 31 18:21:33.103433 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:33.103404    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bfc6c5d55-h9bc5"
Oct 31 18:21:33.279176 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:33.279149    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"]
Oct 31 18:21:33.317445 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:21:33.317415    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56dbaf32_7147_4622_afef_37fb8375bd4f.slice/crio-371db7710fe8c8e1301e0a5e818c3f9350f4772a9b3e684da271c4a1cddc6a26 WatchSource:0}: Error finding container 371db7710fe8c8e1301e0a5e818c3f9350f4772a9b3e684da271c4a1cddc6a26: Status 404 returned error can't find the container with id 371db7710fe8c8e1301e0a5e818c3f9350f4772a9b3e684da271c4a1cddc6a26
Oct 31 18:21:33.432457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:33.432429    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:33.617888 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:33.617819    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:33.801626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:33.801591    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:34.173741 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:34.173705    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx" event={"ID":"56dbaf32-7147-4622-afef-37fb8375bd4f","Type":"ContainerStarted","Data":"371db7710fe8c8e1301e0a5e818c3f9350f4772a9b3e684da271c4a1cddc6a26"}
Oct 31 18:21:36.011244 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:36.011205    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:36.180261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:36.180227    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:36.181958 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:36.181932    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx" event={"ID":"56dbaf32-7147-4622-afef-37fb8375bd4f","Type":"ContainerStarted","Data":"e046718cb01b3696d3c91e8d25b19138ed3f18a336d53dda64bd0286a8bff7f6"}
Oct 31 18:21:36.182123 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:36.182102    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"
Oct 31 18:21:36.186790 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:36.186770    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx"
Oct 31 18:21:36.296588 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:36.296511    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nj5kx" podStartSLOduration=2.152052673 podStartE2EDuration="4.296499802s" totalImagesPullingTime="2.144447129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:36.295691943 +0000 UTC m=+42.442141818" watchObservedRunningTime="2026-10-31 18:21:36.296499802 +0000 UTC m=+42.442949663"
Oct 31 18:21:36.803239 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:36.803199    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:36.973900 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:36.973867    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:37.013461 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.013438    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 18:21:37.016039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.016019    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.069849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.069786    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 18:21:37.069964 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.069939    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 18:21:37.070020 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.070007    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 18:21:37.070063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.070006    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 18:21:37.120025 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.120002    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 18:21:37.120025 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.120021    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1kqh859k72575\""
Oct 31 18:21:37.120235 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.120009    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 18:21:37.160362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160333    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160509 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160366    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160509 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160385    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160509 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160443    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160623 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160517    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160623 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160543    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmhfs\" (UniqueName: \"kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-kube-api-access-gmhfs\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160623 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160558    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160623 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160572    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160623 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160589    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160629    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160719    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160738    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160767    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-config\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160790    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160831    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.160910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160881    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.161022 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160929    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.161022 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160976    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.161022 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.160996    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.161022 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.161016    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.171661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.171623    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 18:21:37.171661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.171630    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 18:21:37.171856 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.171710    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 18:21:37.171856 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.171738    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 18:21:37.171856 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.171744    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 18:21:37.171856 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.171777    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 18:21:37.171856 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.171801    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2zm85\""
Oct 31 18:21:37.221547 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.221522    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 18:21:37.221948 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.221931    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 18:21:37.222058 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.222043    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 18:21:37.223469 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.223453    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 18:21:37.261360 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.261337    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmhfs\" (UniqueName: \"kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-kube-api-access-gmhfs\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.261452 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.261375    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.261452 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.261392    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.261592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.261529    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.261592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.261570    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.261691 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.261611    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.261691 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.261636    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.261853 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.261831    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-config\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.261996 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.261964    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262001    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262028    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262054    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262061    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262080    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262114    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262152    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262189    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262190    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262215    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262248    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262276    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262306    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.262887 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.262398    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.264999 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.264702    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.264999 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.264756    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.265394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.265367    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.265502 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.265401    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.265841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.265816    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.266146 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.266085    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.266219 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.266085    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.266794 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.266767    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.266961 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.266932    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-config\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.267044 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.266942    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.267243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.267217    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.267396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.267375    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.267462 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.267428    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.267716 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.267699    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.268085 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.268059    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.268149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.268107    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.320650 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.320596    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmhfs\" (UniqueName: \"kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-kube-api-access-gmhfs\") pod \"prometheus-k8s-0\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.331443 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.331416    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:37.757163 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:37.757141    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 18:21:37.759421 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:21:37.759391    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16061c6c_512e_4335_b392_c9d34a7d62d6.slice/crio-781e22c3e1c79c08ad5f53712c95e010b673bf42885c8b914b4989d6aff8c802 WatchSource:0}: Error finding container 781e22c3e1c79c08ad5f53712c95e010b673bf42885c8b914b4989d6aff8c802: Status 404 returned error can't find the container with id 781e22c3e1c79c08ad5f53712c95e010b673bf42885c8b914b4989d6aff8c802
Oct 31 18:21:38.190885 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:38.190794    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerStarted","Data":"781e22c3e1c79c08ad5f53712c95e010b673bf42885c8b914b4989d6aff8c802"}
Oct 31 18:21:39.195857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:39.195831    4484 generic.go:356] "Generic (PLEG): container finished" podID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerID="35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0" exitCode=0
Oct 31 18:21:39.196194 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:39.195877    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerDied","Data":"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"}
Oct 31 18:21:40.328178 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:40.328141    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:40.530264 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:40.530222    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:40.686024 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:40.685942    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:41.686123 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:41.686090    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:41.909905 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:41.909868    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:42.603792 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:42.603760    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vlnmf-hb4xw"]
Oct 31 18:21:42.604150 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:42.604121    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-vlnmf-hb4xw" podUID="ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" containerName="insights-gathering" containerID="cri-o://b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b" gracePeriod=30
Oct 31 18:21:42.812022 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:42.811990    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:42.847574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:42.847554    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:21:42.908073 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:42.908046    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-kube-api-access-2v2s4\" (UniqueName: \"kubernetes.io/projected/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-kube-api-access-2v2s4\") pod \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") "
Oct 31 18:21:42.908073 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:42.908082    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-trusted-ca-bundle\") pod \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") "
Oct 31 18:21:42.908297 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:42.908120    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-service-ca-bundle\") pod \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") "
Oct 31 18:21:42.908297 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:42.908157    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-archives-path\") pod \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\" (UID: \"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657\") "
Oct 31 18:21:42.908463 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:42.908437    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-trusted-ca-bundle" pod "ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" (UID: "ff5fb89e-04c5-4f69-ac9b-a12e9e14d657"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:21:42.908557 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:42.908491    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-service-ca-bundle" pod "ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" (UID: "ff5fb89e-04c5-4f69-ac9b-a12e9e14d657"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:21:42.908557 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:42.908521    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-archives-path" pod "ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" (UID: "ff5fb89e-04c5-4f69-ac9b-a12e9e14d657"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:21:42.910568 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:42.910539    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-kube-api-access-2v2s4" pod "ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" (UID: "ff5fb89e-04c5-4f69-ac9b-a12e9e14d657"). InnerVolumeSpecName "kube-api-access-2v2s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:21:43.009325 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.009295    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v2s4\" (UniqueName: \"kubernetes.io/projected/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-kube-api-access-2v2s4\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:21:43.009325 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.009321    4484 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-trusted-ca-bundle\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:21:43.009557 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.009340    4484 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-service-ca-bundle\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:21:43.009557 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.009350    4484 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657-archives-path\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:21:43.019807 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.019777    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:43.212137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.212046    4484 generic.go:356] "Generic (PLEG): container finished" podID="ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" containerID="b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b" exitCode=2
Oct 31 18:21:43.212137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.212088    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vlnmf-hb4xw"
Oct 31 18:21:43.212341 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.212134    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vlnmf-hb4xw" event={"ID":"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657","Type":"ContainerDied","Data":"b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b"}
Oct 31 18:21:43.212341 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.212184    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vlnmf-hb4xw" event={"ID":"ff5fb89e-04c5-4f69-ac9b-a12e9e14d657","Type":"ContainerDied","Data":"91b208cc2de4062418d3cebdc99d47d8a42f3fb382db917c2949fd2df9f0c523"}
Oct 31 18:21:43.212341 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.212204    4484 scope.go:122] "RemoveContainer" containerID="b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b"
Oct 31 18:21:43.216799 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.216773    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerStarted","Data":"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"}
Oct 31 18:21:43.216914 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.216855    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerStarted","Data":"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"}
Oct 31 18:21:43.221766 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.221748    4484 scope.go:122] "RemoveContainer" containerID="b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b"
Oct 31 18:21:43.222058 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:21:43.222038    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b\": container with ID starting with b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b not found: ID does not exist" containerID="b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b"
Oct 31 18:21:43.222106 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.222068    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b"} err="failed to get container status \"b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b\": rpc error: code = NotFound desc = could not find container \"b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b\": container with ID starting with b89fadc1b2a3902b27a1e4a65b40d17b5df9d785b52b8b83255167bf8a56ab9b not found: ID does not exist"
Oct 31 18:21:43.376922 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.376888    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vlnmf-hb4xw"]
Oct 31 18:21:43.385807 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.385775    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-vlnmf-hb4xw"]
Oct 31 18:21:43.989983 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:43.989948    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" path="/var/lib/kubelet/pods/ff5fb89e-04c5-4f69-ac9b-a12e9e14d657/volumes"
Oct 31 18:21:44.525029 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:44.524992    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:44.689234 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:44.689194    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:45.295507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:45.295389    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:45.449495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:45.449436    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:46.231970 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:46.231930    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerStarted","Data":"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"}
Oct 31 18:21:46.232137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:46.232001    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerStarted","Data":"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"}
Oct 31 18:21:46.232137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:46.232034    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerStarted","Data":"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"}
Oct 31 18:21:46.232137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:46.232058    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerStarted","Data":"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"}
Oct 31 18:21:46.508757 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:46.508705    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.060725155 podStartE2EDuration="10.508690242s" totalImagesPullingTime="7.427326704s" totalInitContainerRuntime="20.638383ms" isStatefulPod=false podCreationTimestamp="2026-10-31 18:21:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:21:46.457034066 +0000 UTC m=+52.603483930" watchObservedRunningTime="2026-10-31 18:21:46.508690242 +0000 UTC m=+52.655140104"
Oct 31 18:21:47.332557 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:47.332525    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:49.623446 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:49.623402    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:49.783625 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:49.783588    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:54.185969 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:54.185927    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:54.336950 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:54.336908    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:55.237695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:55.237662    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:55.391088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:55.391053    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:55.467362 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:21:55.467323    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:21:59.034327 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:59.034295    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:59.236131 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:59.236092    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:59.540955 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:59.540918    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:59.690753 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:59.690717    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:21:59.947992 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:59.947889    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:21:59.967130 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:21:59.967102    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:00.295501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:00.295461    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:13.908207 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:13.908173    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:14.054399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:14.054366    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:19.566839 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:19.566800    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:19.726695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:19.726656    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:20.346357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:20.346315    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:21.655400 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:21.655371    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:21.805529 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:21.805460    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:25.469192 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:22:25.469157    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:22:27.597571 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:27.597531    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:27.803688 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:27.803657    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:32.840833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:32.840775    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 18:22:32.842186 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:32.842116    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c" gracePeriod=600
Oct 31 18:22:32.844003 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:32.843941    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="kube-rbac-proxy" containerID="cri-o://7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400" gracePeriod=600
Oct 31 18:22:32.844982 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:32.844961    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="thanos-sidecar" containerID="cri-o://08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82" gracePeriod=600
Oct 31 18:22:32.845635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:32.845184    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="config-reloader" containerID="cri-o://f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09" gracePeriod=600
Oct 31 18:22:32.846611 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:32.844901    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="kube-rbac-proxy-web" containerID="cri-o://b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d" gracePeriod=600
Oct 31 18:22:32.846919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:32.846551    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="prometheus" containerID="cri-o://03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869" gracePeriod=600
Oct 31 18:22:33.100751 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.100694    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.212362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212324    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-1\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212363    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-2\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212656 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212395    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-thanos-prometheus-http-client-file\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212656 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212427    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212656 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212552    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-web-config\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-web-config\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212656 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212594    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-kube-rbac-proxy\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212656 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212630    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212664    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-serving-certs-ca-bundle\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212704    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-tls-assets\" (UniqueName: \"kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-tls-assets\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212740    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-db\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212773    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-kube-api-access-gmhfs\" (UniqueName: \"kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-kube-api-access-gmhfs\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212802    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-metrics-client-ca\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212818    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-2" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:33.212907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212831    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-1" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:33.212907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212845    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-grpc-tls\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.212907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212878    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-config\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-config\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.213346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212905    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-tls\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.213346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212956    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-0\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.213346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.212991    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-config-out\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.213346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.213022    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-metrics-client-certs\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.213346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.213051    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-trusted-ca-bundle\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.213346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.213097    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-kubelet-serving-ca-bundle\") pod \"16061c6c-512e-4335-b392-c9d34a7d62d6\" (UID: \"16061c6c-512e-4335-b392-c9d34a7d62d6\") "
Oct 31 18:22:33.213346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.213295    4484 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.213346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.213312    4484 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.213991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.213637    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-kubelet-serving-ca-bundle" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:33.213991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.213940    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-metrics-client-ca" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:33.214498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.214110    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-db" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:22:33.215662 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.215513    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-thanos-prometheus-http-client-file" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:33.215662 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.215533    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-kube-rbac-proxy-web" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:33.215662 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.215588    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-tls-assets" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:22:33.216408 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.215925    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-0" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:33.216408 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.216205    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-serving-certs-ca-bundle" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:33.216602 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.216523    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-grpc-tls" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:33.216665 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.216594    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-kube-api-access-gmhfs" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "kube-api-access-gmhfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:22:33.217047 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.216856    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-kube-rbac-proxy" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:33.217047 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.216941    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-trusted-ca-bundle" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:22:33.217813 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.217788    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-thanos-sidecar-tls" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:33.217965 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.217941    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-config" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:33.218272 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.218256    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-config-out" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:22:33.219023 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.219000    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-metrics-client-certs" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:33.219117 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.219084    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-tls" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:33.226524 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.226500    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-web-config" pod "16061c6c-512e-4335-b392-c9d34a7d62d6" (UID: "16061c6c-512e-4335-b392-c9d34a7d62d6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:22:33.313812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313779    4484 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.313812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313808    4484 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.313812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313820    4484 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-web-config\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313830    4484 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-kube-rbac-proxy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313840    4484 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313852    4484 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313865    4484 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-tls-assets\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313874    4484 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-db\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313882    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmhfs\" (UniqueName: \"kubernetes.io/projected/16061c6c-512e-4335-b392-c9d34a7d62d6-kube-api-access-gmhfs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313892    4484 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-metrics-client-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313901    4484 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-grpc-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313909    4484 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-config\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313917    4484 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313929    4484 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313941    4484 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16061c6c-512e-4335-b392-c9d34a7d62d6-config-out\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313949    4484 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16061c6c-512e-4335-b392-c9d34a7d62d6-secret-metrics-client-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313958    4484 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.314100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.313966    4484 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16061c6c-512e-4335-b392-c9d34a7d62d6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:22:33.395529 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395420    4484 generic.go:356] "Generic (PLEG): container finished" podID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerID="5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c" exitCode=0
Oct 31 18:22:33.395529 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395447    4484 generic.go:356] "Generic (PLEG): container finished" podID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerID="7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400" exitCode=0
Oct 31 18:22:33.395529 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395455    4484 generic.go:356] "Generic (PLEG): container finished" podID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerID="b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d" exitCode=0
Oct 31 18:22:33.395529 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395461    4484 generic.go:356] "Generic (PLEG): container finished" podID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerID="08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82" exitCode=0
Oct 31 18:22:33.395529 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395466    4484 generic.go:356] "Generic (PLEG): container finished" podID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerID="f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09" exitCode=0
Oct 31 18:22:33.395529 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395498    4484 generic.go:356] "Generic (PLEG): container finished" podID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerID="03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869" exitCode=0
Oct 31 18:22:33.395529 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395512    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerDied","Data":"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"}
Oct 31 18:22:33.395968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395526    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.395968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395634    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerDied","Data":"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"}
Oct 31 18:22:33.395968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395677    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerDied","Data":"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"}
Oct 31 18:22:33.395968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395693    4484 scope.go:122] "RemoveContainer" containerID="5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"
Oct 31 18:22:33.395968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395713    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerDied","Data":"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"}
Oct 31 18:22:33.395968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395753    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerDied","Data":"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"}
Oct 31 18:22:33.395968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395792    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerDied","Data":"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"}
Oct 31 18:22:33.395968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.395832    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16061c6c-512e-4335-b392-c9d34a7d62d6","Type":"ContainerDied","Data":"781e22c3e1c79c08ad5f53712c95e010b673bf42885c8b914b4989d6aff8c802"}
Oct 31 18:22:33.403297 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.403119    4484 scope.go:122] "RemoveContainer" containerID="7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"
Oct 31 18:22:33.410410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.410390    4484 scope.go:122] "RemoveContainer" containerID="b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"
Oct 31 18:22:33.417261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.417241    4484 scope.go:122] "RemoveContainer" containerID="08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"
Oct 31 18:22:33.423983 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.423955    4484 scope.go:122] "RemoveContainer" containerID="f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"
Oct 31 18:22:33.431124 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.431103    4484 scope.go:122] "RemoveContainer" containerID="03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"
Oct 31 18:22:33.438985 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.438966    4484 scope.go:122] "RemoveContainer" containerID="35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"
Oct 31 18:22:33.445970 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.445940    4484 scope.go:122] "RemoveContainer" containerID="5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"
Oct 31 18:22:33.446263 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:22:33.446244    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": container with ID starting with 5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c not found: ID does not exist" containerID="5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"
Oct 31 18:22:33.446312 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.446274    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"} err="failed to get container status \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": rpc error: code = NotFound desc = could not find container \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": container with ID starting with 5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c not found: ID does not exist"
Oct 31 18:22:33.446312 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.446292    4484 scope.go:122] "RemoveContainer" containerID="7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"
Oct 31 18:22:33.446556 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:22:33.446537    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": container with ID starting with 7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400 not found: ID does not exist" containerID="7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"
Oct 31 18:22:33.446630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.446567    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"} err="failed to get container status \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": rpc error: code = NotFound desc = could not find container \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": container with ID starting with 7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400 not found: ID does not exist"
Oct 31 18:22:33.446630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.446589    4484 scope.go:122] "RemoveContainer" containerID="b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"
Oct 31 18:22:33.446835 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:22:33.446820    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": container with ID starting with b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d not found: ID does not exist" containerID="b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"
Oct 31 18:22:33.446873 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.446840    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"} err="failed to get container status \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": rpc error: code = NotFound desc = could not find container \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": container with ID starting with b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d not found: ID does not exist"
Oct 31 18:22:33.446873 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.446854    4484 scope.go:122] "RemoveContainer" containerID="08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"
Oct 31 18:22:33.447065 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:22:33.447050    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": container with ID starting with 08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82 not found: ID does not exist" containerID="08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"
Oct 31 18:22:33.447111 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.447069    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"} err="failed to get container status \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": rpc error: code = NotFound desc = could not find container \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": container with ID starting with 08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82 not found: ID does not exist"
Oct 31 18:22:33.447111 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.447083    4484 scope.go:122] "RemoveContainer" containerID="f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"
Oct 31 18:22:33.447268 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:22:33.447253    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": container with ID starting with f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09 not found: ID does not exist" containerID="f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"
Oct 31 18:22:33.447310 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.447271    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"} err="failed to get container status \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": rpc error: code = NotFound desc = could not find container \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": container with ID starting with f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09 not found: ID does not exist"
Oct 31 18:22:33.447310 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.447281    4484 scope.go:122] "RemoveContainer" containerID="03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"
Oct 31 18:22:33.447494 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:22:33.447460    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": container with ID starting with 03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869 not found: ID does not exist" containerID="03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"
Oct 31 18:22:33.447541 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.447493    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"} err="failed to get container status \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": rpc error: code = NotFound desc = could not find container \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": container with ID starting with 03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869 not found: ID does not exist"
Oct 31 18:22:33.447541 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.447509    4484 scope.go:122] "RemoveContainer" containerID="35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"
Oct 31 18:22:33.447721 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:22:33.447708    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": container with ID starting with 35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0 not found: ID does not exist" containerID="35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"
Oct 31 18:22:33.447765 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.447723    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"} err="failed to get container status \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": rpc error: code = NotFound desc = could not find container \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": container with ID starting with 35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0 not found: ID does not exist"
Oct 31 18:22:33.447765 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.447734    4484 scope.go:122] "RemoveContainer" containerID="5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"
Oct 31 18:22:33.447923 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.447909    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"} err="failed to get container status \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": rpc error: code = NotFound desc = could not find container \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": container with ID starting with 5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c not found: ID does not exist"
Oct 31 18:22:33.447923 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.447923    4484 scope.go:122] "RemoveContainer" containerID="7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"
Oct 31 18:22:33.448104 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.448090    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"} err="failed to get container status \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": rpc error: code = NotFound desc = could not find container \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": container with ID starting with 7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400 not found: ID does not exist"
Oct 31 18:22:33.448104 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.448103    4484 scope.go:122] "RemoveContainer" containerID="b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"
Oct 31 18:22:33.448275 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.448258    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"} err="failed to get container status \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": rpc error: code = NotFound desc = could not find container \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": container with ID starting with b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d not found: ID does not exist"
Oct 31 18:22:33.448315 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.448276    4484 scope.go:122] "RemoveContainer" containerID="08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"
Oct 31 18:22:33.448506 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.448489    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"} err="failed to get container status \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": rpc error: code = NotFound desc = could not find container \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": container with ID starting with 08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82 not found: ID does not exist"
Oct 31 18:22:33.448506 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.448505    4484 scope.go:122] "RemoveContainer" containerID="f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"
Oct 31 18:22:33.448715 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.448691    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"} err="failed to get container status \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": rpc error: code = NotFound desc = could not find container \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": container with ID starting with f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09 not found: ID does not exist"
Oct 31 18:22:33.448754 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.448717    4484 scope.go:122] "RemoveContainer" containerID="03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"
Oct 31 18:22:33.448941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.448924    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"} err="failed to get container status \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": rpc error: code = NotFound desc = could not find container \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": container with ID starting with 03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869 not found: ID does not exist"
Oct 31 18:22:33.448985 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.448942    4484 scope.go:122] "RemoveContainer" containerID="35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"
Oct 31 18:22:33.449187 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.449164    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"} err="failed to get container status \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": rpc error: code = NotFound desc = could not find container \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": container with ID starting with 35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0 not found: ID does not exist"
Oct 31 18:22:33.449240 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.449189    4484 scope.go:122] "RemoveContainer" containerID="5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"
Oct 31 18:22:33.449518 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.449491    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"} err="failed to get container status \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": rpc error: code = NotFound desc = could not find container \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": container with ID starting with 5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c not found: ID does not exist"
Oct 31 18:22:33.449518 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.449517    4484 scope.go:122] "RemoveContainer" containerID="7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"
Oct 31 18:22:33.449767 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.449749    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"} err="failed to get container status \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": rpc error: code = NotFound desc = could not find container \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": container with ID starting with 7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400 not found: ID does not exist"
Oct 31 18:22:33.449843 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.449769    4484 scope.go:122] "RemoveContainer" containerID="b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"
Oct 31 18:22:33.450018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.449997    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"} err="failed to get container status \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": rpc error: code = NotFound desc = could not find container \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": container with ID starting with b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d not found: ID does not exist"
Oct 31 18:22:33.450069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.450020    4484 scope.go:122] "RemoveContainer" containerID="08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"
Oct 31 18:22:33.450262 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.450245    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"} err="failed to get container status \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": rpc error: code = NotFound desc = could not find container \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": container with ID starting with 08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82 not found: ID does not exist"
Oct 31 18:22:33.450318 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.450264    4484 scope.go:122] "RemoveContainer" containerID="f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"
Oct 31 18:22:33.450500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.450467    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"} err="failed to get container status \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": rpc error: code = NotFound desc = could not find container \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": container with ID starting with f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09 not found: ID does not exist"
Oct 31 18:22:33.450565 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.450501    4484 scope.go:122] "RemoveContainer" containerID="03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"
Oct 31 18:22:33.450772 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.450752    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"} err="failed to get container status \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": rpc error: code = NotFound desc = could not find container \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": container with ID starting with 03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869 not found: ID does not exist"
Oct 31 18:22:33.450772 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.450771    4484 scope.go:122] "RemoveContainer" containerID="35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"
Oct 31 18:22:33.450999 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.450980    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"} err="failed to get container status \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": rpc error: code = NotFound desc = could not find container \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": container with ID starting with 35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0 not found: ID does not exist"
Oct 31 18:22:33.451049 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.451001    4484 scope.go:122] "RemoveContainer" containerID="5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"
Oct 31 18:22:33.451210 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.451193    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"} err="failed to get container status \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": rpc error: code = NotFound desc = could not find container \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": container with ID starting with 5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c not found: ID does not exist"
Oct 31 18:22:33.451249 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.451212    4484 scope.go:122] "RemoveContainer" containerID="7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"
Oct 31 18:22:33.451422 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.451407    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"} err="failed to get container status \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": rpc error: code = NotFound desc = could not find container \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": container with ID starting with 7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400 not found: ID does not exist"
Oct 31 18:22:33.451482 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.451423    4484 scope.go:122] "RemoveContainer" containerID="b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"
Oct 31 18:22:33.451670 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.451653    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"} err="failed to get container status \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": rpc error: code = NotFound desc = could not find container \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": container with ID starting with b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d not found: ID does not exist"
Oct 31 18:22:33.451723 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.451670    4484 scope.go:122] "RemoveContainer" containerID="08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"
Oct 31 18:22:33.451870 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.451854    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"} err="failed to get container status \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": rpc error: code = NotFound desc = could not find container \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": container with ID starting with 08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82 not found: ID does not exist"
Oct 31 18:22:33.451911 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.451870    4484 scope.go:122] "RemoveContainer" containerID="f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"
Oct 31 18:22:33.452093 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.452076    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"} err="failed to get container status \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": rpc error: code = NotFound desc = could not find container \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": container with ID starting with f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09 not found: ID does not exist"
Oct 31 18:22:33.452136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.452094    4484 scope.go:122] "RemoveContainer" containerID="03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"
Oct 31 18:22:33.452300 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.452284    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"} err="failed to get container status \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": rpc error: code = NotFound desc = could not find container \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": container with ID starting with 03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869 not found: ID does not exist"
Oct 31 18:22:33.452342 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.452301    4484 scope.go:122] "RemoveContainer" containerID="35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"
Oct 31 18:22:33.452516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.452499    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"} err="failed to get container status \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": rpc error: code = NotFound desc = could not find container \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": container with ID starting with 35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0 not found: ID does not exist"
Oct 31 18:22:33.452561 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.452517    4484 scope.go:122] "RemoveContainer" containerID="5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"
Oct 31 18:22:33.452718 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.452700    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"} err="failed to get container status \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": rpc error: code = NotFound desc = could not find container \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": container with ID starting with 5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c not found: ID does not exist"
Oct 31 18:22:33.452761 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.452719    4484 scope.go:122] "RemoveContainer" containerID="7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"
Oct 31 18:22:33.452917 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.452897    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"} err="failed to get container status \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": rpc error: code = NotFound desc = could not find container \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": container with ID starting with 7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400 not found: ID does not exist"
Oct 31 18:22:33.452956 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.452916    4484 scope.go:122] "RemoveContainer" containerID="b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"
Oct 31 18:22:33.453098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.453081    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"} err="failed to get container status \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": rpc error: code = NotFound desc = could not find container \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": container with ID starting with b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d not found: ID does not exist"
Oct 31 18:22:33.453141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.453098    4484 scope.go:122] "RemoveContainer" containerID="08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"
Oct 31 18:22:33.453306 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.453289    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"} err="failed to get container status \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": rpc error: code = NotFound desc = could not find container \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": container with ID starting with 08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82 not found: ID does not exist"
Oct 31 18:22:33.453351 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.453306    4484 scope.go:122] "RemoveContainer" containerID="f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"
Oct 31 18:22:33.453532 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.453513    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"} err="failed to get container status \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": rpc error: code = NotFound desc = could not find container \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": container with ID starting with f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09 not found: ID does not exist"
Oct 31 18:22:33.453575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.453532    4484 scope.go:122] "RemoveContainer" containerID="03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"
Oct 31 18:22:33.453748 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.453722    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"} err="failed to get container status \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": rpc error: code = NotFound desc = could not find container \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": container with ID starting with 03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869 not found: ID does not exist"
Oct 31 18:22:33.453832 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.453748    4484 scope.go:122] "RemoveContainer" containerID="35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"
Oct 31 18:22:33.453965 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.453947    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"} err="failed to get container status \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": rpc error: code = NotFound desc = could not find container \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": container with ID starting with 35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0 not found: ID does not exist"
Oct 31 18:22:33.454009 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.453967    4484 scope.go:122] "RemoveContainer" containerID="5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"
Oct 31 18:22:33.454170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.454153    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c"} err="failed to get container status \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": rpc error: code = NotFound desc = could not find container \"5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c\": container with ID starting with 5ec88ffdf69e7852a6a17f7b3ce30e3d34c1b9f93f4bbebf304e90e438cd208c not found: ID does not exist"
Oct 31 18:22:33.454212 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.454170    4484 scope.go:122] "RemoveContainer" containerID="7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"
Oct 31 18:22:33.454389 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.454369    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400"} err="failed to get container status \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": rpc error: code = NotFound desc = could not find container \"7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400\": container with ID starting with 7cc7d7c0571799a3f2799ac1936ccec56252164caaae18eaa0a18ef1fe79d400 not found: ID does not exist"
Oct 31 18:22:33.454437 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.454390    4484 scope.go:122] "RemoveContainer" containerID="b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"
Oct 31 18:22:33.454606 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.454588    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d"} err="failed to get container status \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": rpc error: code = NotFound desc = could not find container \"b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d\": container with ID starting with b55af7b02d98a7ea3500b25b3d2c53921bc88ab346ffd0fc63d9caafe8ac472d not found: ID does not exist"
Oct 31 18:22:33.454665 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.454606    4484 scope.go:122] "RemoveContainer" containerID="08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"
Oct 31 18:22:33.454837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.454822    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82"} err="failed to get container status \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": rpc error: code = NotFound desc = could not find container \"08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82\": container with ID starting with 08fe259d1b877f91c3ff0a700a5381544f5bd33186c667cadc7e13a1633c8e82 not found: ID does not exist"
Oct 31 18:22:33.454881 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.454839    4484 scope.go:122] "RemoveContainer" containerID="f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"
Oct 31 18:22:33.455054 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.455037    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09"} err="failed to get container status \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": rpc error: code = NotFound desc = could not find container \"f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09\": container with ID starting with f6a2301cdcd057c4113e31b4e6468a1d9b3eeaeef6f292cf81ba65082681fc09 not found: ID does not exist"
Oct 31 18:22:33.455094 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.455054    4484 scope.go:122] "RemoveContainer" containerID="03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"
Oct 31 18:22:33.455251 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.455233    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869"} err="failed to get container status \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": rpc error: code = NotFound desc = could not find container \"03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869\": container with ID starting with 03b04cde8fb2b5eec69a3e0edc234a73b6ea094a9d041f80b8d45e2860db7869 not found: ID does not exist"
Oct 31 18:22:33.455293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.455252    4484 scope.go:122] "RemoveContainer" containerID="35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"
Oct 31 18:22:33.455450 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.455431    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0"} err="failed to get container status \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": rpc error: code = NotFound desc = could not find container \"35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0\": container with ID starting with 35fec2daaea889c48fb736aa8a987876aa0382470a3dd47a768a659c8b26f8d0 not found: ID does not exist"
Oct 31 18:22:33.563671 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.563640    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 18:22:33.568770 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.568746    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 18:22:33.593285 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593259    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 18:22:33.593547 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593535    4484 cpu_manager.go:414] "containerMap: removing container" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="config-reloader"
Oct 31 18:22:33.593595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593549    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="config-reloader"
Oct 31 18:22:33.593595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593559    4484 cpu_manager.go:414] "containerMap: removing container" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="prometheus"
Oct 31 18:22:33.593595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593564    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="prometheus"
Oct 31 18:22:33.593595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593570    4484 cpu_manager.go:414] "containerMap: removing container" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="kube-rbac-proxy"
Oct 31 18:22:33.593595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593576    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="kube-rbac-proxy"
Oct 31 18:22:33.593595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593581    4484 cpu_manager.go:414] "containerMap: removing container" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="kube-rbac-proxy-thanos"
Oct 31 18:22:33.593595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593585    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="kube-rbac-proxy-thanos"
Oct 31 18:22:33.593595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593599    4484 cpu_manager.go:414] "containerMap: removing container" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="thanos-sidecar"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593603    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="thanos-sidecar"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593610    4484 cpu_manager.go:414] "containerMap: removing container" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="kube-rbac-proxy-web"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593615    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="kube-rbac-proxy-web"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593623    4484 cpu_manager.go:414] "containerMap: removing container" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="init-config-reloader"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593627    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="init-config-reloader"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593636    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" containerName="insights-gathering"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593639    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" containerName="insights-gathering"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593673    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff5fb89e-04c5-4f69-ac9b-a12e9e14d657" containerName="insights-gathering"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593682    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="config-reloader"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593688    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="prometheus"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593692    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="kube-rbac-proxy"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593697    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="kube-rbac-proxy-thanos"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593703    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="thanos-sidecar"
Oct 31 18:22:33.593818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.593708    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" containerName="kube-rbac-proxy-web"
Oct 31 18:22:33.596731 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.596714    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.653424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.653331    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 18:22:33.653785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.653716    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 18:22:33.653785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.653727    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 18:22:33.653785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.653774    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 18:22:33.654072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.653852    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 18:22:33.654072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.653919    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2zm85\""
Oct 31 18:22:33.654072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.653933    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 18:22:33.654072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.653951    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 18:22:33.654072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.653961    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1kqh859k72575\""
Oct 31 18:22:33.654072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.653970    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 18:22:33.654072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.653975    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 18:22:33.654072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.654005    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 18:22:33.654357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.654080    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 18:22:33.654357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.654153    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 18:22:33.654661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.654646    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 18:22:33.665063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.665035    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 18:22:33.705117 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.705089    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 18:22:33.716537 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.716510    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.716609 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.716541    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.716609 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.716558    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.716673 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.716633    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.716710 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.716678    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.716710 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.716699    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.716768 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.716754    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.716799 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.716775    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-web-config\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.716799 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.716793    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.716885 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.716814    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57064eeb-d684-49f8-8eec-f85bf72412d0-config-out\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.716964 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.716943    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.717022 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.716980    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.717063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.717027    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.717063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.717051    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.717148 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.717068    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576sm\" (UniqueName: \"kubernetes.io/projected/57064eeb-d684-49f8-8eec-f85bf72412d0-kube-api-access-576sm\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.717148 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.717086    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57064eeb-d684-49f8-8eec-f85bf72412d0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.717148 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.717108    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.717235 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.717187    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-config\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.717235 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.717205    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.717235 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.717220    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.813565 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.813533    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 18:22:33.818371 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.818337    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.818535 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.818383    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.818535 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.818412    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.818535 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.818440    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.818535 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.818465    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.818535 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.818517    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.818766 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.818721    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819032    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-web-config\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819092    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819122    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57064eeb-d684-49f8-8eec-f85bf72412d0-config-out\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819181    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819187    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819209    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819250    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819286    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819315    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576sm\" (UniqueName: \"kubernetes.io/projected/57064eeb-d684-49f8-8eec-f85bf72412d0-kube-api-access-576sm\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819327    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819347    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57064eeb-d684-49f8-8eec-f85bf72412d0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819377    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819376    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819440    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-config\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.819466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819497    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.821758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819524    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.821758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.819676    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.821758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.820103    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.821758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.820103    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.821758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.820936    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.823069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.823042    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/57064eeb-d684-49f8-8eec-f85bf72412d0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.824515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.824109    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.824515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.824172    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.824515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.824387    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.824515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.824432    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.824515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.824464    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.824753 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.824554    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-web-config\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.824925 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.824904    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/57064eeb-d684-49f8-8eec-f85bf72412d0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.824968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.824905    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.825376 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.825353    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/57064eeb-d684-49f8-8eec-f85bf72412d0-config-out\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.825735 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.825710    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-config\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.825858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.825840    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/57064eeb-d684-49f8-8eec-f85bf72412d0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.878700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.878674    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-576sm\" (UniqueName: \"kubernetes.io/projected/57064eeb-d684-49f8-8eec-f85bf72412d0-kube-api-access-576sm\") pod \"prometheus-k8s-0\" (UID: \"57064eeb-d684-49f8-8eec-f85bf72412d0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.920618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.920524    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:33.991462 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:33.991426    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16061c6c-512e-4335-b392-c9d34a7d62d6" path="/var/lib/kubelet/pods/16061c6c-512e-4335-b392-c9d34a7d62d6/volumes"
Oct 31 18:22:34.346868 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:34.346678    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 18:22:34.349423 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:22:34.349386    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57064eeb_d684_49f8_8eec_f85bf72412d0.slice/crio-bcaf80488597b28393d19cb35405601dc3e845bc38ec8c438b5394f473ab0825 WatchSource:0}: Error finding container bcaf80488597b28393d19cb35405601dc3e845bc38ec8c438b5394f473ab0825: Status 404 returned error can't find the container with id bcaf80488597b28393d19cb35405601dc3e845bc38ec8c438b5394f473ab0825
Oct 31 18:22:34.400578 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:34.400549    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57064eeb-d684-49f8-8eec-f85bf72412d0","Type":"ContainerStarted","Data":"bcaf80488597b28393d19cb35405601dc3e845bc38ec8c438b5394f473ab0825"}
Oct 31 18:22:35.144284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:35.144249    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:35.405615 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:35.405529    4484 generic.go:356] "Generic (PLEG): container finished" podID="57064eeb-d684-49f8-8eec-f85bf72412d0" containerID="1ed8c5b9ac61d0a5631f1e9792f5564fa2ba318a729d1621dd74cc214b89cc6c" exitCode=0
Oct 31 18:22:35.405615 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:35.405596    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57064eeb-d684-49f8-8eec-f85bf72412d0","Type":"ContainerDied","Data":"1ed8c5b9ac61d0a5631f1e9792f5564fa2ba318a729d1621dd74cc214b89cc6c"}
Oct 31 18:22:36.412783 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:36.412753    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57064eeb-d684-49f8-8eec-f85bf72412d0","Type":"ContainerStarted","Data":"bfc14786b9168edf094d50a82bd113f4df3fa0175ce237bdcf0a7cc5904a376f"}
Oct 31 18:22:36.413253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:36.412812    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57064eeb-d684-49f8-8eec-f85bf72412d0","Type":"ContainerStarted","Data":"5c3dd0907e7da32e53e0bbad481a8f5ea61c6eff514518c1f6c61af6a5114da3"}
Oct 31 18:22:36.413253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:36.412841    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57064eeb-d684-49f8-8eec-f85bf72412d0","Type":"ContainerStarted","Data":"5bfc20582692d166ac50fb76896087bb54631de21c18409c88ca02446a70b3e9"}
Oct 31 18:22:36.413253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:36.412865    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57064eeb-d684-49f8-8eec-f85bf72412d0","Type":"ContainerStarted","Data":"08f33dbfb6ea453f951d2ff4302800ebfdf0ff28b3c2ca6c27896c1fef236847"}
Oct 31 18:22:36.413253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:36.412886    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57064eeb-d684-49f8-8eec-f85bf72412d0","Type":"ContainerStarted","Data":"5ccb67d5e4a3218f55abde3ee3309ba9c58797a5b4c5c88d00befafdb53dfb59"}
Oct 31 18:22:36.413253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:36.412910    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"57064eeb-d684-49f8-8eec-f85bf72412d0","Type":"ContainerStarted","Data":"67c9beb70bb5a1c1f800f9cfc1a402840d0e04655b363d2a41f271cbf2b8a13b"}
Oct 31 18:22:36.537773 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:36.537722    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.519259875 podStartE2EDuration="3.537708012s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.448137ms" isStatefulPod=false podCreationTimestamp="2026-10-31 18:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:22:36.536453797 +0000 UTC m=+102.682903695" watchObservedRunningTime="2026-10-31 18:22:36.537708012 +0000 UTC m=+102.684157908"
Oct 31 18:22:38.921295 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:38.921260    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:22:40.703068 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:40.703025    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:44.537238 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:44.537189    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:44.742956 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:22:44.742918    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:22:55.468023 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:22:55.467979    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:23:25.467180 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:23:25.467145    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:23:25.471736 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:23:25.471707    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:23:33.921417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:23:33.921382    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:23:33.937223 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:23:33.937199    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:23:34.629944 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:23:34.629917    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 18:23:41.014079 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:23:41.014039    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:23:55.466858 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:23:55.466821    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:24:25.467284 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:24:25.467240    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:24:33.208048 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:33.208013    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:33.367175 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:33.367141    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:36.410042 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:36.409961    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:36.627136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:36.627101    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:37.775572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:37.775532    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:37.947685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:37.947644    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:38.871930 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:38.871891    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:41.358752 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:41.358719    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:44.182071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:44.182035    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:44.394720 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:44.394684    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:44.732222 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:44.732184    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:44.900625 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:44.900590    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:45.049494 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:45.049441    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:48.758254 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:24:48.758218    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:24:55.467019 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:24:55.466975    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:25:22.437276 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:22.437241    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:23.266575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.266543    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"]
Oct 31 18:25:23.269833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.269813    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.317218 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.317189    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"]
Oct 31 18:25:23.323046 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.323024    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 18:25:23.323212 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.323098    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 18:25:23.323212 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.323155    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 18:25:23.358940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.358907    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 18:25:23.358940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.358925    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-k5llp\""
Oct 31 18:25:23.359301 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.359283    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 18:25:23.452207 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.452177    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9856447f-af40-42cf-ad0d-00d240995c00-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.452207 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.452210    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9856447f-af40-42cf-ad0d-00d240995c00-manifests\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.452707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.452245    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9856447f-af40-42cf-ad0d-00d240995c00-metrics-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.452707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.452366    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9856447f-af40-42cf-ad0d-00d240995c00-tmp\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.452707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.452398    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7lgt\" (UniqueName: \"kubernetes.io/projected/9856447f-af40-42cf-ad0d-00d240995c00-kube-api-access-b7lgt\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.452707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.452420    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9856447f-af40-42cf-ad0d-00d240995c00-webhook-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.553386 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.553294    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9856447f-af40-42cf-ad0d-00d240995c00-tmp\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.553386 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.553344    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7lgt\" (UniqueName: \"kubernetes.io/projected/9856447f-af40-42cf-ad0d-00d240995c00-kube-api-access-b7lgt\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.553386 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.553374    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9856447f-af40-42cf-ad0d-00d240995c00-webhook-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.553690 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.553425    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9856447f-af40-42cf-ad0d-00d240995c00-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.553690 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.553563    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9856447f-af40-42cf-ad0d-00d240995c00-manifests\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.553690 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.553629    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9856447f-af40-42cf-ad0d-00d240995c00-metrics-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.553981 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.553951    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9856447f-af40-42cf-ad0d-00d240995c00-manifests\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.556089 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.556054    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9856447f-af40-42cf-ad0d-00d240995c00-metrics-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.556198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.556061    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9856447f-af40-42cf-ad0d-00d240995c00-webhook-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.556198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.556168    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9856447f-af40-42cf-ad0d-00d240995c00-tmp\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.556334 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.556318    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9856447f-af40-42cf-ad0d-00d240995c00-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.613017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.612986    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7lgt\" (UniqueName: \"kubernetes.io/projected/9856447f-af40-42cf-ad0d-00d240995c00-kube-api-access-b7lgt\") pod \"opendatahub-operator-controller-manager-dd8c87985-qqkrq\" (UID: \"9856447f-af40-42cf-ad0d-00d240995c00\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:23.884725 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:23.884632    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:24.097542 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:24.097500    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:24.429989 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:24.429949    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:24.480992 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:24.480956    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"]
Oct 31 18:25:24.483100 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:25:24.483073    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9856447f_af40_42cf_ad0d_00d240995c00.slice/crio-baeaed0c169df77dfc1275b3e23f15c2b4c5df16d697e1c4e60e63953153a354 WatchSource:0}: Error finding container baeaed0c169df77dfc1275b3e23f15c2b4c5df16d697e1c4e60e63953153a354: Status 404 returned error can't find the container with id baeaed0c169df77dfc1275b3e23f15c2b4c5df16d697e1c4e60e63953153a354
Oct 31 18:25:24.614512 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:24.614462    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:24.974966 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:24.974932    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq" event={"ID":"9856447f-af40-42cf-ad0d-00d240995c00","Type":"ContainerStarted","Data":"baeaed0c169df77dfc1275b3e23f15c2b4c5df16d697e1c4e60e63953153a354"}
Oct 31 18:25:25.472704 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:25:25.472666    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:25:26.089933 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:26.089876    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:26.254089 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:26.254049    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:26.890119 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:26.890076    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:27.052365 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:27.052312    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:28.993949 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:28.993915    4484 generic.go:356] "Generic (PLEG): container finished" podID="9856447f-af40-42cf-ad0d-00d240995c00" containerID="e038089dd76ab15a9de803ac310c24272b1e7b7110846e0a544365e5a6b620f1" exitCode=0
Oct 31 18:25:28.994396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:28.993965    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq" event={"ID":"9856447f-af40-42cf-ad0d-00d240995c00","Type":"ContainerDied","Data":"e038089dd76ab15a9de803ac310c24272b1e7b7110846e0a544365e5a6b620f1"}
Oct 31 18:25:29.999079 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:29.999046    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq" event={"ID":"9856447f-af40-42cf-ad0d-00d240995c00","Type":"ContainerStarted","Data":"c110ecba4c233a49a8b8b39d1a7e74ff5b5bb04a76dfafa73b99b3617d0396bc"}
Oct 31 18:25:29.999495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:29.999150    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:30.232203 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:30.232101    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq" podStartSLOduration=1.968107216 podStartE2EDuration="7.232085482s" totalImagesPullingTime="4.827789638s" totalInitContainerRuntime="436.188628ms" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:30.181814672 +0000 UTC m=+276.328264549" watchObservedRunningTime="2026-10-31 18:25:30.232085482 +0000 UTC m=+276.378535343"
Oct 31 18:25:31.008801 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:31.008765    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:32.230065 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:32.230026    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:37.017704 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:37.017669    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:38.254205 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:38.254170    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:38.655655 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:38.655570    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:38.867416 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:38.867377    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:40.321366 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:40.321329    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:40.842031 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:40.841998    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh"]
Oct 31 18:25:40.846415 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:40.846396    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh"
Oct 31 18:25:40.881499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:40.879763    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr6w7\" (UniqueName: \"kubernetes.io/projected/9a6a03ed-d418-4666-b617-e41ec4b753f5-kube-api-access-xr6w7\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh\" (UID: \"9a6a03ed-d418-4666-b617-e41ec4b753f5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh"
Oct 31 18:25:40.881499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:40.879820    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a6a03ed-d418-4666-b617-e41ec4b753f5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh\" (UID: \"9a6a03ed-d418-4666-b617-e41ec4b753f5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh"
Oct 31 18:25:40.891286 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:40.891254    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh"]
Oct 31 18:25:40.940623 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:40.940592    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 18:25:40.940772 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:40.940644    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 18:25:40.940772 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:40.940651    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-k27z6\""
Oct 31 18:25:40.981176 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:40.981133    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr6w7\" (UniqueName: \"kubernetes.io/projected/9a6a03ed-d418-4666-b617-e41ec4b753f5-kube-api-access-xr6w7\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh\" (UID: \"9a6a03ed-d418-4666-b617-e41ec4b753f5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh"
Oct 31 18:25:40.981176 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:40.981177    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a6a03ed-d418-4666-b617-e41ec4b753f5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh\" (UID: \"9a6a03ed-d418-4666-b617-e41ec4b753f5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh"
Oct 31 18:25:40.981644 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:40.981621    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a6a03ed-d418-4666-b617-e41ec4b753f5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh\" (UID: \"9a6a03ed-d418-4666-b617-e41ec4b753f5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh"
Oct 31 18:25:41.004591 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:41.004562    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-dd8c87985-qqkrq"
Oct 31 18:25:41.044254 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:41.044230    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr6w7\" (UniqueName: \"kubernetes.io/projected/9a6a03ed-d418-4666-b617-e41ec4b753f5-kube-api-access-xr6w7\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh\" (UID: \"9a6a03ed-d418-4666-b617-e41ec4b753f5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh"
Oct 31 18:25:41.156051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:41.155985    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh"
Oct 31 18:25:41.295418 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:41.295390    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:41.582086 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:41.582056    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh"]
Oct 31 18:25:41.583439 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:25:41.583416    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6a03ed_d418_4666_b617_e41ec4b753f5.slice/crio-1b8b88b1d70f903a3a84b0fad09f2be90b70d7c50ead8211d60e1c80a667fa33 WatchSource:0}: Error finding container 1b8b88b1d70f903a3a84b0fad09f2be90b70d7c50ead8211d60e1c80a667fa33: Status 404 returned error can't find the container with id 1b8b88b1d70f903a3a84b0fad09f2be90b70d7c50ead8211d60e1c80a667fa33
Oct 31 18:25:41.693866 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:41.693840    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:42.037337 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:42.037307    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh" event={"ID":"9a6a03ed-d418-4666-b617-e41ec4b753f5","Type":"ContainerStarted","Data":"1b8b88b1d70f903a3a84b0fad09f2be90b70d7c50ead8211d60e1c80a667fa33"}
Oct 31 18:25:42.652130 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:42.652086    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:43.847865 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:43.847828    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:45.841893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:45.841807    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:46.054540 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:46.054504    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh" event={"ID":"9a6a03ed-d418-4666-b617-e41ec4b753f5","Type":"ContainerStarted","Data":"048f3fc67da46f94fae52feb7ed80176ef997555fc532d28ae1223ce09653ee4"}
Oct 31 18:25:46.475378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:46.475326    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:47.041662 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:47.041626    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:47.431858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:47.431770    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:49.010110 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:49.010078    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:49.929253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:49.929223    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:49.943899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:49.943860    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jnqbh" podStartSLOduration=6.014848407 podStartE2EDuration="9.943846689s" totalImagesPullingTime="3.928998282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:46.312779736 +0000 UTC m=+292.459229602" watchObservedRunningTime="2026-10-31 18:25:49.943846689 +0000 UTC m=+296.090296553"
Oct 31 18:25:49.944483 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:49.944456    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6"]
Oct 31 18:25:49.947658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:49.947645    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6"
Oct 31 18:25:50.010508 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.010467    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6"]
Oct 31 18:25:50.054646 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.054607    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qzzb\" (UniqueName: \"kubernetes.io/projected/716e8c5b-b2c0-4338-9d9b-9dbb76ed820b-kube-api-access-9qzzb\") pod \"cert-manager-cainjector-5c57bbd4fc-gmzj6\" (UID: \"716e8c5b-b2c0-4338-9d9b-9dbb76ed820b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6"
Oct 31 18:25:50.054796 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.054674    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/716e8c5b-b2c0-4338-9d9b-9dbb76ed820b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-gmzj6\" (UID: \"716e8c5b-b2c0-4338-9d9b-9dbb76ed820b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6"
Oct 31 18:25:50.059614 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.059598    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 18:25:50.059724 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.059708    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 18:25:50.059782 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.059773    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-h8qjz\""
Oct 31 18:25:50.156014 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.155963    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/716e8c5b-b2c0-4338-9d9b-9dbb76ed820b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-gmzj6\" (UID: \"716e8c5b-b2c0-4338-9d9b-9dbb76ed820b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6"
Oct 31 18:25:50.156195 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.156042    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qzzb\" (UniqueName: \"kubernetes.io/projected/716e8c5b-b2c0-4338-9d9b-9dbb76ed820b-kube-api-access-9qzzb\") pod \"cert-manager-cainjector-5c57bbd4fc-gmzj6\" (UID: \"716e8c5b-b2c0-4338-9d9b-9dbb76ed820b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6"
Oct 31 18:25:50.188202 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.188126    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:50.213863 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.213841    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/716e8c5b-b2c0-4338-9d9b-9dbb76ed820b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-gmzj6\" (UID: \"716e8c5b-b2c0-4338-9d9b-9dbb76ed820b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6"
Oct 31 18:25:50.215557 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.215535    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qzzb\" (UniqueName: \"kubernetes.io/projected/716e8c5b-b2c0-4338-9d9b-9dbb76ed820b-kube-api-access-9qzzb\") pod \"cert-manager-cainjector-5c57bbd4fc-gmzj6\" (UID: \"716e8c5b-b2c0-4338-9d9b-9dbb76ed820b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6"
Oct 31 18:25:50.270707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.270675    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6"
Oct 31 18:25:50.645554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:50.645526    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6"]
Oct 31 18:25:50.648110 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:25:50.648072    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod716e8c5b_b2c0_4338_9d9b_9dbb76ed820b.slice/crio-1879fe0583571966f6cdf3e8b9f21c3e3d8c1dc85a8c44ce95941d9f498d40b1 WatchSource:0}: Error finding container 1879fe0583571966f6cdf3e8b9f21c3e3d8c1dc85a8c44ce95941d9f498d40b1: Status 404 returned error can't find the container with id 1879fe0583571966f6cdf3e8b9f21c3e3d8c1dc85a8c44ce95941d9f498d40b1
Oct 31 18:25:51.073160 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:51.073128    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6" event={"ID":"716e8c5b-b2c0-4338-9d9b-9dbb76ed820b","Type":"ContainerStarted","Data":"1879fe0583571966f6cdf3e8b9f21c3e3d8c1dc85a8c44ce95941d9f498d40b1"}
Oct 31 18:25:51.490014 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:51.489981    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:53.030740 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:53.030701    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:53.223987 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:53.223949    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:53.645813 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:53.645779    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:53.868257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:53.868221    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:54.006107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:54.006085    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:25:54.006901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:54.006883    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:25:54.806353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:54.806321    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"]
Oct 31 18:25:54.811158 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:54.811135    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"
Oct 31 18:25:54.866148 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:54.866116    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-lt49b\""
Oct 31 18:25:54.867381 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:54.867356    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"]
Oct 31 18:25:54.894126 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:54.894108    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzqfk\" (UniqueName: \"kubernetes.io/projected/f4769ed5-7f1a-43fc-a9ec-260207cbb2d7-kube-api-access-wzqfk\") pod \"cert-manager-webhook-5dbbbf75cc-wrcbh\" (UID: \"f4769ed5-7f1a-43fc-a9ec-260207cbb2d7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"
Oct 31 18:25:54.894190 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:54.894142    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4769ed5-7f1a-43fc-a9ec-260207cbb2d7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wrcbh\" (UID: \"f4769ed5-7f1a-43fc-a9ec-260207cbb2d7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"
Oct 31 18:25:54.995254 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:54.995221    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzqfk\" (UniqueName: \"kubernetes.io/projected/f4769ed5-7f1a-43fc-a9ec-260207cbb2d7-kube-api-access-wzqfk\") pod \"cert-manager-webhook-5dbbbf75cc-wrcbh\" (UID: \"f4769ed5-7f1a-43fc-a9ec-260207cbb2d7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"
Oct 31 18:25:54.995254 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:54.995258    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4769ed5-7f1a-43fc-a9ec-260207cbb2d7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wrcbh\" (UID: \"f4769ed5-7f1a-43fc-a9ec-260207cbb2d7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"
Oct 31 18:25:55.053376 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:55.053336    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4769ed5-7f1a-43fc-a9ec-260207cbb2d7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wrcbh\" (UID: \"f4769ed5-7f1a-43fc-a9ec-260207cbb2d7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"
Oct 31 18:25:55.053547 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:55.053399    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzqfk\" (UniqueName: \"kubernetes.io/projected/f4769ed5-7f1a-43fc-a9ec-260207cbb2d7-kube-api-access-wzqfk\") pod \"cert-manager-webhook-5dbbbf75cc-wrcbh\" (UID: \"f4769ed5-7f1a-43fc-a9ec-260207cbb2d7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"
Oct 31 18:25:55.091595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:55.091523    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6" event={"ID":"716e8c5b-b2c0-4338-9d9b-9dbb76ed820b","Type":"ContainerStarted","Data":"fcc893fad21dec02e8626f62add54c0ccc3b860770eb67018928416968b9ce03"}
Oct 31 18:25:55.123213 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:55.123177    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"
Oct 31 18:25:55.206972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:55.206925    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-gmzj6" podStartSLOduration=1.963877445 podStartE2EDuration="6.206913163s" totalImagesPullingTime="4.243035718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:55.206343058 +0000 UTC m=+301.352792923" watchObservedRunningTime="2026-10-31 18:25:55.206913163 +0000 UTC m=+301.353363023"
Oct 31 18:25:55.467339 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:25:55.467259    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:25:55.486428 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:55.486406    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"]
Oct 31 18:25:55.488932 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:25:55.488906    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4769ed5_7f1a_43fc_a9ec_260207cbb2d7.slice/crio-fc2189efa58f937f686b3ade169a2f3d311510568f16ec662b374c989c9a1ee8 WatchSource:0}: Error finding container fc2189efa58f937f686b3ade169a2f3d311510568f16ec662b374c989c9a1ee8: Status 404 returned error can't find the container with id fc2189efa58f937f686b3ade169a2f3d311510568f16ec662b374c989c9a1ee8
Oct 31 18:25:56.097551 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:56.097515    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh" event={"ID":"f4769ed5-7f1a-43fc-a9ec-260207cbb2d7","Type":"ContainerStarted","Data":"9eea80572d1f196755c3b311f164ab3c5842624ef2db4c9fe5a53101d7a870e3"}
Oct 31 18:25:56.098002 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:56.097559    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh" event={"ID":"f4769ed5-7f1a-43fc-a9ec-260207cbb2d7","Type":"ContainerStarted","Data":"fc2189efa58f937f686b3ade169a2f3d311510568f16ec662b374c989c9a1ee8"}
Oct 31 18:25:56.098002 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:56.097681    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"
Oct 31 18:25:56.212804 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:56.212752    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh" podStartSLOduration=2.212740811 podStartE2EDuration="2.212740811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:56.211789811 +0000 UTC m=+302.358239672" watchObservedRunningTime="2026-10-31 18:25:56.212740811 +0000 UTC m=+302.359190675"
Oct 31 18:25:56.512995 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:56.512958    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:56.680687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:56.680648    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:57.632121 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:57.632086    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:57.791441 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:57.791404    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:58.131159 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.131122    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:58.181305 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.181277    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-6gd55"]
Oct 31 18:25:58.184182 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.184166    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-6gd55"
Oct 31 18:25:58.224867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.224837    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkkgx\" (UniqueName: \"kubernetes.io/projected/7a4345db-2094-48f8-a726-5f3023b08533-kube-api-access-vkkgx\") pod \"cert-manager-597b95cb84-6gd55\" (UID: \"7a4345db-2094-48f8-a726-5f3023b08533\") " pod="cert-manager/cert-manager-597b95cb84-6gd55"
Oct 31 18:25:58.225006 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.224870    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a4345db-2094-48f8-a726-5f3023b08533-bound-sa-token\") pod \"cert-manager-597b95cb84-6gd55\" (UID: \"7a4345db-2094-48f8-a726-5f3023b08533\") " pod="cert-manager/cert-manager-597b95cb84-6gd55"
Oct 31 18:25:58.240335 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.240312    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-6gd55"]
Oct 31 18:25:58.288426 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.288395    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:58.289461 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.289444    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ptz4m\""
Oct 31 18:25:58.325720 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.325694    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkkgx\" (UniqueName: \"kubernetes.io/projected/7a4345db-2094-48f8-a726-5f3023b08533-kube-api-access-vkkgx\") pod \"cert-manager-597b95cb84-6gd55\" (UID: \"7a4345db-2094-48f8-a726-5f3023b08533\") " pod="cert-manager/cert-manager-597b95cb84-6gd55"
Oct 31 18:25:58.325897 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.325731    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a4345db-2094-48f8-a726-5f3023b08533-bound-sa-token\") pod \"cert-manager-597b95cb84-6gd55\" (UID: \"7a4345db-2094-48f8-a726-5f3023b08533\") " pod="cert-manager/cert-manager-597b95cb84-6gd55"
Oct 31 18:25:58.384406 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.384335    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a4345db-2094-48f8-a726-5f3023b08533-bound-sa-token\") pod \"cert-manager-597b95cb84-6gd55\" (UID: \"7a4345db-2094-48f8-a726-5f3023b08533\") " pod="cert-manager/cert-manager-597b95cb84-6gd55"
Oct 31 18:25:58.384406 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.384397    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkkgx\" (UniqueName: \"kubernetes.io/projected/7a4345db-2094-48f8-a726-5f3023b08533-kube-api-access-vkkgx\") pod \"cert-manager-597b95cb84-6gd55\" (UID: \"7a4345db-2094-48f8-a726-5f3023b08533\") " pod="cert-manager/cert-manager-597b95cb84-6gd55"
Oct 31 18:25:58.495601 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.495565    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-6gd55"
Oct 31 18:25:58.866251 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:58.866226    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-6gd55"]
Oct 31 18:25:58.868448 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:25:58.868418    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a4345db_2094_48f8_a726_5f3023b08533.slice/crio-ccd6e7eec1e03b2b8499aed5955391da8dbf481a3bc9ff4826fe0bed4e67ff74 WatchSource:0}: Error finding container ccd6e7eec1e03b2b8499aed5955391da8dbf481a3bc9ff4826fe0bed4e67ff74: Status 404 returned error can't find the container with id ccd6e7eec1e03b2b8499aed5955391da8dbf481a3bc9ff4826fe0bed4e67ff74
Oct 31 18:25:59.110552 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.110515    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-6gd55" event={"ID":"7a4345db-2094-48f8-a726-5f3023b08533","Type":"ContainerStarted","Data":"5449581c7d6043d024ae4dab91a743728cd6031be49392a36a2cf9f0bf943dc9"}
Oct 31 18:25:59.110552 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.110560    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-6gd55" event={"ID":"7a4345db-2094-48f8-a726-5f3023b08533","Type":"ContainerStarted","Data":"ccd6e7eec1e03b2b8499aed5955391da8dbf481a3bc9ff4826fe0bed4e67ff74"}
Oct 31 18:25:59.223808 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.223697    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-6gd55" podStartSLOduration=1.223684202 podStartE2EDuration="1.223684202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:25:59.223419839 +0000 UTC m=+305.369869704" watchObservedRunningTime="2026-10-31 18:25:59.223684202 +0000 UTC m=+305.370134066"
Oct 31 18:25:59.661924 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.661886    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"]
Oct 31 18:25:59.665210 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.665187    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"
Oct 31 18:25:59.674835 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.674812    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:59.720074 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.720049    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 18:25:59.737253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.737212    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70aa0a3-26be-42df-881e-2d9f6a58697d-tmp\") pod \"kube-auth-proxy-5d4f97d94f-8v994\" (UID: \"a70aa0a3-26be-42df-881e-2d9f6a58697d\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"
Oct 31 18:25:59.737410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.737287    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a70aa0a3-26be-42df-881e-2d9f6a58697d-tls-certs\") pod \"kube-auth-proxy-5d4f97d94f-8v994\" (UID: \"a70aa0a3-26be-42df-881e-2d9f6a58697d\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"
Oct 31 18:25:59.737455 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.737430    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5jhn\" (UniqueName: \"kubernetes.io/projected/a70aa0a3-26be-42df-881e-2d9f6a58697d-kube-api-access-w5jhn\") pod \"kube-auth-proxy-5d4f97d94f-8v994\" (UID: \"a70aa0a3-26be-42df-881e-2d9f6a58697d\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"
Oct 31 18:25:59.770878 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.770818    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 18:25:59.770878 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.770836    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 18:25:59.770878 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.770870    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 18:25:59.771138 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.770886    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hzmzn\""
Oct 31 18:25:59.772722 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.772700    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"]
Oct 31 18:25:59.838685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.838651    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5jhn\" (UniqueName: \"kubernetes.io/projected/a70aa0a3-26be-42df-881e-2d9f6a58697d-kube-api-access-w5jhn\") pod \"kube-auth-proxy-5d4f97d94f-8v994\" (UID: \"a70aa0a3-26be-42df-881e-2d9f6a58697d\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"
Oct 31 18:25:59.838685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.838690    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70aa0a3-26be-42df-881e-2d9f6a58697d-tmp\") pod \"kube-auth-proxy-5d4f97d94f-8v994\" (UID: \"a70aa0a3-26be-42df-881e-2d9f6a58697d\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"
Oct 31 18:25:59.838943 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.838739    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a70aa0a3-26be-42df-881e-2d9f6a58697d-tls-certs\") pod \"kube-auth-proxy-5d4f97d94f-8v994\" (UID: \"a70aa0a3-26be-42df-881e-2d9f6a58697d\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"
Oct 31 18:25:59.841043 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.841013    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70aa0a3-26be-42df-881e-2d9f6a58697d-tmp\") pod \"kube-auth-proxy-5d4f97d94f-8v994\" (UID: \"a70aa0a3-26be-42df-881e-2d9f6a58697d\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"
Oct 31 18:25:59.841280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.841262    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a70aa0a3-26be-42df-881e-2d9f6a58697d-tls-certs\") pod \"kube-auth-proxy-5d4f97d94f-8v994\" (UID: \"a70aa0a3-26be-42df-881e-2d9f6a58697d\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"
Oct 31 18:25:59.894371 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.894337    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:25:59.896530 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.896504    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5jhn\" (UniqueName: \"kubernetes.io/projected/a70aa0a3-26be-42df-881e-2d9f6a58697d-kube-api-access-w5jhn\") pod \"kube-auth-proxy-5d4f97d94f-8v994\" (UID: \"a70aa0a3-26be-42df-881e-2d9f6a58697d\") " pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"
Oct 31 18:25:59.977318 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:25:59.977250    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"
Oct 31 18:26:00.203887 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:00.203842    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:00.342746 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:00.342716    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994"]
Oct 31 18:26:00.345209 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:26:00.345181    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70aa0a3_26be_42df_881e_2d9f6a58697d.slice/crio-66a424c2ce42cb6da9c05c60acff6d3adad1cf9af34264335cbd37c128a201b0 WatchSource:0}: Error finding container 66a424c2ce42cb6da9c05c60acff6d3adad1cf9af34264335cbd37c128a201b0: Status 404 returned error can't find the container with id 66a424c2ce42cb6da9c05c60acff6d3adad1cf9af34264335cbd37c128a201b0
Oct 31 18:26:00.347204 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:00.347181    4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:26:00.385815 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:00.385789    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:00.593146 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:00.593061    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:01.118948 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:01.118912    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994" event={"ID":"a70aa0a3-26be-42df-881e-2d9f6a58697d","Type":"ContainerStarted","Data":"66a424c2ce42cb6da9c05c60acff6d3adad1cf9af34264335cbd37c128a201b0"}
Oct 31 18:26:01.843204 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:01.843110    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:02.027093 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:02.027058    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:02.104785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:02.104703    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wrcbh"
Oct 31 18:26:05.137060 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:05.136971    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994" event={"ID":"a70aa0a3-26be-42df-881e-2d9f6a58697d","Type":"ContainerStarted","Data":"02914cbcc665103200a6de25980cca06c8c479aa94b6aea2e94277bbbbd2a07a"}
Oct 31 18:26:05.252828 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:05.252765    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d4f97d94f-8v994" podStartSLOduration=1.792206584 podStartE2EDuration="6.252750873s" totalImagesPullingTime="4.460544289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:26:05.252284475 +0000 UTC m=+311.398734339" watchObservedRunningTime="2026-10-31 18:26:05.252750873 +0000 UTC m=+311.399200745"
Oct 31 18:26:06.556845 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:06.556764    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:06.703793 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:06.703758    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:07.259488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.259440    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:07.633528 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.633418    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:07.692391 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.692356    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-np6dt"]
Oct 31 18:26:07.695423 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.695407    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt"
Oct 31 18:26:07.800778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.800736    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 18:26:07.800778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.800757    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 18:26:07.801416 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.800864    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cfcnz\""
Oct 31 18:26:07.801416 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.800937    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 18:26:07.802207 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.802177    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-np6dt"]
Oct 31 18:26:07.813249 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.813222    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7pc5\" (UniqueName: \"kubernetes.io/projected/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-kube-api-access-k7pc5\") pod \"limitador-limitador-74dfcfb894-np6dt\" (UID: \"84f65d7f-8f5a-45f8-a02f-37e4543dadc7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt"
Oct 31 18:26:07.813417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.813394    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-config-file\") pod \"limitador-limitador-74dfcfb894-np6dt\" (UID: \"84f65d7f-8f5a-45f8-a02f-37e4543dadc7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt"
Oct 31 18:26:07.894324 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.894223    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:07.914216 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.914191    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-config-file\") pod \"limitador-limitador-74dfcfb894-np6dt\" (UID: \"84f65d7f-8f5a-45f8-a02f-37e4543dadc7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt"
Oct 31 18:26:07.914385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.914235    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7pc5\" (UniqueName: \"kubernetes.io/projected/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-kube-api-access-k7pc5\") pod \"limitador-limitador-74dfcfb894-np6dt\" (UID: \"84f65d7f-8f5a-45f8-a02f-37e4543dadc7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt"
Oct 31 18:26:07.914979 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.914953    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-config-file\") pod \"limitador-limitador-74dfcfb894-np6dt\" (UID: \"84f65d7f-8f5a-45f8-a02f-37e4543dadc7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt"
Oct 31 18:26:07.933237 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.933207    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-np6dt"]
Oct 31 18:26:07.972781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:07.972753    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7pc5\" (UniqueName: \"kubernetes.io/projected/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-kube-api-access-k7pc5\") pod \"limitador-limitador-74dfcfb894-np6dt\" (UID: \"84f65d7f-8f5a-45f8-a02f-37e4543dadc7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt"
Oct 31 18:26:08.007740 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:08.007712    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt"
Oct 31 18:26:08.052279 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:08.052242    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:08.376640 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:08.376613    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-np6dt"]
Oct 31 18:26:08.379107 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:26:08.379081    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f65d7f_8f5a_45f8_a02f_37e4543dadc7.slice/crio-1f3b04a5f636ff7d9eb42569aa9bf275f25038f6fbf31c1b4af5181bf3688bee WatchSource:0}: Error finding container 1f3b04a5f636ff7d9eb42569aa9bf275f25038f6fbf31c1b4af5181bf3688bee: Status 404 returned error can't find the container with id 1f3b04a5f636ff7d9eb42569aa9bf275f25038f6fbf31c1b4af5181bf3688bee
Oct 31 18:26:09.153161 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:09.153127    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt" event={"ID":"84f65d7f-8f5a-45f8-a02f-37e4543dadc7","Type":"ContainerStarted","Data":"1f3b04a5f636ff7d9eb42569aa9bf275f25038f6fbf31c1b4af5181bf3688bee"}
Oct 31 18:26:09.219193 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:09.219151    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:09.413141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:09.413064    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:11.855557 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:11.855529    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:26:11.859000 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:11.858981    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb"
Oct 31 18:26:11.948779 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:11.948685    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:26:11.952923 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:11.952887    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67ql\" (UniqueName: \"kubernetes.io/projected/5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a-kube-api-access-l67ql\") pod \"limitador-limitador-7c9b87d574-z57jb\" (UID: \"5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb"
Oct 31 18:26:11.953098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:11.953065    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a-config-file\") pod \"limitador-limitador-7c9b87d574-z57jb\" (UID: \"5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb"
Oct 31 18:26:12.008707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:12.008678    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:12.053573 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:12.053551    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a-config-file\") pod \"limitador-limitador-7c9b87d574-z57jb\" (UID: \"5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb"
Oct 31 18:26:12.053760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:12.053589    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67ql\" (UniqueName: \"kubernetes.io/projected/5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a-kube-api-access-l67ql\") pod \"limitador-limitador-7c9b87d574-z57jb\" (UID: \"5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb"
Oct 31 18:26:12.054315 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:12.054292    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a-config-file\") pod \"limitador-limitador-7c9b87d574-z57jb\" (UID: \"5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb"
Oct 31 18:26:12.109963 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:12.109908    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67ql\" (UniqueName: \"kubernetes.io/projected/5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a-kube-api-access-l67ql\") pod \"limitador-limitador-7c9b87d574-z57jb\" (UID: \"5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb"
Oct 31 18:26:12.156213 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:12.156182    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:26:12.166377 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:12.166350    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt" event={"ID":"84f65d7f-8f5a-45f8-a02f-37e4543dadc7","Type":"ContainerStarted","Data":"4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f"}
Oct 31 18:26:12.166513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:12.166498    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt"
Oct 31 18:26:12.173724 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:12.173706    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb"
Oct 31 18:26:12.214352 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:12.214319    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:12.283025 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:12.282976    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt" podStartSLOduration=1.61337445 podStartE2EDuration="5.28296212s" totalImagesPullingTime="3.66958767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:26:12.281923733 +0000 UTC m=+318.428373601" watchObservedRunningTime="2026-10-31 18:26:12.28296212 +0000 UTC m=+318.429412087"
Oct 31 18:26:12.540237 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:12.540188    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:26:12.542415 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:26:12.542386    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb6bbc4_2e5f_4b4d_9146_70c70cb8891a.slice/crio-ddaa7b0f057e907838d9c973a55478b3e8ae58355c2957bf6c3bf9512446e137 WatchSource:0}: Error finding container ddaa7b0f057e907838d9c973a55478b3e8ae58355c2957bf6c3bf9512446e137: Status 404 returned error can't find the container with id ddaa7b0f057e907838d9c973a55478b3e8ae58355c2957bf6c3bf9512446e137
Oct 31 18:26:13.171239 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:13.171198    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb" event={"ID":"5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a","Type":"ContainerStarted","Data":"64eeac499220cba87a7f181302107154567291911e8c82a3121f88b3af279fc5"}
Oct 31 18:26:13.171694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:13.171255    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb" event={"ID":"5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a","Type":"ContainerStarted","Data":"ddaa7b0f057e907838d9c973a55478b3e8ae58355c2957bf6c3bf9512446e137"}
Oct 31 18:26:13.171694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:13.171396    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb"
Oct 31 18:26:13.285488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:13.285419    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb" podStartSLOduration=2.28540677 podStartE2EDuration="2.28540677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:26:13.284864718 +0000 UTC m=+319.431314583" watchObservedRunningTime="2026-10-31 18:26:13.28540677 +0000 UTC m=+319.431856676"
Oct 31 18:26:14.382765 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:14.382731    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:14.539126 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:14.539095    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:14.980309 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:14.980279    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:15.136605 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:15.136559    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:17.480173 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:17.480139    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:17.667023 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:17.666976    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:19.826572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:19.826526    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:20.037196 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:20.037163    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:20.531986 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:20.531950    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:20.686015 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:20.685985    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:20.850872 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:20.850781    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:22.575267 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:22.575233    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:22.765284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:22.765247    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:23.172425 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:23.172397    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt"
Oct 31 18:26:24.175922 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:24.175897    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-z57jb"
Oct 31 18:26:24.341598 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:24.341565    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-np6dt"]
Oct 31 18:26:24.341751 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:24.341729    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt" podUID="84f65d7f-8f5a-45f8-a02f-37e4543dadc7" containerName="limitador" containerID="cri-o://4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f" gracePeriod=30
Oct 31 18:26:24.890306 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:24.890279    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt"
Oct 31 18:26:25.068931 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.068900    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-kube-api-access-k7pc5\" (UniqueName: \"kubernetes.io/projected/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-kube-api-access-k7pc5\") pod \"84f65d7f-8f5a-45f8-a02f-37e4543dadc7\" (UID: \"84f65d7f-8f5a-45f8-a02f-37e4543dadc7\") "
Oct 31 18:26:25.069097 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.068965    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-config-file\" (UniqueName: \"kubernetes.io/configmap/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-config-file\") pod \"84f65d7f-8f5a-45f8-a02f-37e4543dadc7\" (UID: \"84f65d7f-8f5a-45f8-a02f-37e4543dadc7\") "
Oct 31 18:26:25.069366 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.069341    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-config-file" pod "84f65d7f-8f5a-45f8-a02f-37e4543dadc7" (UID: "84f65d7f-8f5a-45f8-a02f-37e4543dadc7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:26:25.071354 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.071332    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-kube-api-access-k7pc5" pod "84f65d7f-8f5a-45f8-a02f-37e4543dadc7" (UID: "84f65d7f-8f5a-45f8-a02f-37e4543dadc7"). InnerVolumeSpecName "kube-api-access-k7pc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:26:25.170337 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.170297    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7pc5\" (UniqueName: \"kubernetes.io/projected/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-kube-api-access-k7pc5\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:26:25.170337 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.170328    4484 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84f65d7f-8f5a-45f8-a02f-37e4543dadc7-config-file\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:26:25.222774 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.222739    4484 generic.go:356] "Generic (PLEG): container finished" podID="84f65d7f-8f5a-45f8-a02f-37e4543dadc7" containerID="4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f" exitCode=0
Oct 31 18:26:25.223191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.222789    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt"
Oct 31 18:26:25.223191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.222814    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt" event={"ID":"84f65d7f-8f5a-45f8-a02f-37e4543dadc7","Type":"ContainerDied","Data":"4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f"}
Oct 31 18:26:25.223191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.222870    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-np6dt" event={"ID":"84f65d7f-8f5a-45f8-a02f-37e4543dadc7","Type":"ContainerDied","Data":"1f3b04a5f636ff7d9eb42569aa9bf275f25038f6fbf31c1b4af5181bf3688bee"}
Oct 31 18:26:25.223191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.222901    4484 scope.go:122] "RemoveContainer" containerID="4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f"
Oct 31 18:26:25.231398 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.231382    4484 scope.go:122] "RemoveContainer" containerID="4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f"
Oct 31 18:26:25.231708 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:26:25.231693    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f\": container with ID starting with 4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f not found: ID does not exist" containerID="4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f"
Oct 31 18:26:25.231747 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.231718    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f"} err="failed to get container status \"4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f\": rpc error: code = NotFound desc = could not find container \"4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f\": container with ID starting with 4919ce69abb7a53fdf0774db33c8f71424c12d36f6b24421138362a5c766207f not found: ID does not exist"
Oct 31 18:26:25.387218 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.387141    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-np6dt"]
Oct 31 18:26:25.437867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.437837    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-np6dt"]
Oct 31 18:26:25.466996 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:26:25.466963    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:26:25.991299 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:25.991264    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f65d7f-8f5a-45f8-a02f-37e4543dadc7" path="/var/lib/kubelet/pods/84f65d7f-8f5a-45f8-a02f-37e4543dadc7/volumes"
Oct 31 18:26:26.339748 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:26.339661    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:26.501991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:26.501951    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:30.287878 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:30.287840    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:30.507324 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:30.507286    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:30.908543 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:30.908503    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"]
Oct 31 18:26:30.908843 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:30.908831    4484 cpu_manager.go:414] "containerMap: removing container" podUID="84f65d7f-8f5a-45f8-a02f-37e4543dadc7" containerName="limitador"
Oct 31 18:26:30.908886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:30.908846    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f65d7f-8f5a-45f8-a02f-37e4543dadc7" containerName="limitador"
Oct 31 18:26:30.908920 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:30.908899    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f65d7f-8f5a-45f8-a02f-37e4543dadc7" containerName="limitador"
Oct 31 18:26:30.911394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:30.911378    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"
Oct 31 18:26:30.967027 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:30.966995    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 18:26:30.968730 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:30.968703    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"]
Oct 31 18:26:31.017198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:31.017169    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 18:26:31.017334 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:31.017294    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/916306e5-63fe-465c-85bf-8957717253de-data\") pod \"postgres-7cdbf6dc44-m59v4\" (UID: \"916306e5-63fe-465c-85bf-8957717253de\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"
Oct 31 18:26:31.017334 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:31.017325    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4wsr\" (UniqueName: \"kubernetes.io/projected/916306e5-63fe-465c-85bf-8957717253de-kube-api-access-p4wsr\") pod \"postgres-7cdbf6dc44-m59v4\" (UID: \"916306e5-63fe-465c-85bf-8957717253de\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"
Oct 31 18:26:31.022119 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:31.022095    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:31.058267 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:31.058239    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 31 18:26:31.058267 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:31.058258    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dvdzr\""
Oct 31 18:26:31.118006 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:31.117977    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4wsr\" (UniqueName: \"kubernetes.io/projected/916306e5-63fe-465c-85bf-8957717253de-kube-api-access-p4wsr\") pod \"postgres-7cdbf6dc44-m59v4\" (UID: \"916306e5-63fe-465c-85bf-8957717253de\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"
Oct 31 18:26:31.118142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:31.118078    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/916306e5-63fe-465c-85bf-8957717253de-data\") pod \"postgres-7cdbf6dc44-m59v4\" (UID: \"916306e5-63fe-465c-85bf-8957717253de\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"
Oct 31 18:26:31.118435 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:31.118418    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/916306e5-63fe-465c-85bf-8957717253de-data\") pod \"postgres-7cdbf6dc44-m59v4\" (UID: \"916306e5-63fe-465c-85bf-8957717253de\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"
Oct 31 18:26:31.176552 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:31.176461    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4wsr\" (UniqueName: \"kubernetes.io/projected/916306e5-63fe-465c-85bf-8957717253de-kube-api-access-p4wsr\") pod \"postgres-7cdbf6dc44-m59v4\" (UID: \"916306e5-63fe-465c-85bf-8957717253de\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"
Oct 31 18:26:31.222661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:31.222633    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"
Oct 31 18:26:31.590927 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:31.590902    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"]
Oct 31 18:26:31.593378 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:26:31.593349    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916306e5_63fe_465c_85bf_8957717253de.slice/crio-c04f8c2ca907c27faee0075f3dd5c6344be462b777398627f35d117384acbbb9 WatchSource:0}: Error finding container c04f8c2ca907c27faee0075f3dd5c6344be462b777398627f35d117384acbbb9: Status 404 returned error can't find the container with id c04f8c2ca907c27faee0075f3dd5c6344be462b777398627f35d117384acbbb9
Oct 31 18:26:32.253094 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:32.253060    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4" event={"ID":"916306e5-63fe-465c-85bf-8957717253de","Type":"ContainerStarted","Data":"c04f8c2ca907c27faee0075f3dd5c6344be462b777398627f35d117384acbbb9"}
Oct 31 18:26:36.880201 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:36.880158    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:37.047304 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:37.047265    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:39.285919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:39.285879    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4" event={"ID":"916306e5-63fe-465c-85bf-8957717253de","Type":"ContainerStarted","Data":"583797bd6ed5bfdcbea1acc6e9391ece45839cb82e68c66d99325c0ee7a404b5"}
Oct 31 18:26:39.286327 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:39.286008    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"
Oct 31 18:26:39.401196 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:39.401142    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4" podStartSLOduration=2.5081767729999997 podStartE2EDuration="9.401122031s" totalImagesPullingTime="6.892945258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:26:39.400317143 +0000 UTC m=+345.546767009" watchObservedRunningTime="2026-10-31 18:26:39.401122031 +0000 UTC m=+345.547571896"
Oct 31 18:26:42.046154 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:42.046110    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:45.320362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:45.320332    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m59v4"
Oct 31 18:26:52.155159 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:52.155121    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:52.394712 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:52.394678    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:52.565043 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:52.565013    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:26:55.467341 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:26:55.467296    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:26:59.873283 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:26:59.873220    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:00.156994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:00.156875    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:02.837543 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.837504    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"]
Oct 31 18:27:02.844564 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.844534    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:02.898532 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.898498    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 31 18:27:02.948903 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.948838    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 18:27:02.948903 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.948846    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 31 18:27:02.949135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.948990    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-tbthl\""
Oct 31 18:27:02.949135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.949033    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 31 18:27:02.949135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.949051    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 18:27:02.950188 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.950166    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"]
Oct 31 18:27:02.989694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.989659    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:02.989873 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.989708    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxkc\" (UniqueName: \"kubernetes.io/projected/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-kube-api-access-kvxkc\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:02.989873 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.989820    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:02.989943 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:02.989879    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:03.091087 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:03.090997    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:03.091087 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:03.091057    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:03.091297 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:03.091101    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxkc\" (UniqueName: \"kubernetes.io/projected/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-kube-api-access-kvxkc\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:03.091297 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:03.091249    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:03.091408 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:27:03.091388    4484 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 31 18:27:03.091531 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:27:03.091519    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-proxy-source podName:9fc7ddc9-d304-433d-9f6a-55d677ec5ad0 nodeName:}" failed. No retries permitted until 2026-10-31 18:27:03.591499303 +0000 UTC m=+369.737949160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-6xxg6" (UID: "9fc7ddc9-d304-433d-9f6a-55d677ec5ad0") : configmap "usage-tenancy-proxy-source" not found
Oct 31 18:27:03.091838 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:03.091820    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:03.093660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:03.093639    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:03.132142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:03.132100    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:03.156650 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:03.156620    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxkc\" (UniqueName: \"kubernetes.io/projected/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-kube-api-access-kvxkc\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:03.370351 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:03.370275    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:03.596946 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:03.596911    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:03.597776 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:03.597756    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9fc7ddc9-d304-433d-9f6a-55d677ec5ad0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-6xxg6\" (UID: \"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:03.756583 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:03.756548    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:04.141461 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:04.141377    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"]
Oct 31 18:27:04.145405 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:27:04.145370    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc7ddc9_d304_433d_9f6a_55d677ec5ad0.slice/crio-546e62acabc43fd221ca4db7dd89b12e04d9b6ce71cc3a6ae1c5b7b199d83396 WatchSource:0}: Error finding container 546e62acabc43fd221ca4db7dd89b12e04d9b6ce71cc3a6ae1c5b7b199d83396: Status 404 returned error can't find the container with id 546e62acabc43fd221ca4db7dd89b12e04d9b6ce71cc3a6ae1c5b7b199d83396
Oct 31 18:27:04.393130 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:04.393039    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6" event={"ID":"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0","Type":"ContainerStarted","Data":"546e62acabc43fd221ca4db7dd89b12e04d9b6ce71cc3a6ae1c5b7b199d83396"}
Oct 31 18:27:18.454457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:18.454368    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6" event={"ID":"9fc7ddc9-d304-433d-9f6a-55d677ec5ad0","Type":"ContainerStarted","Data":"051fcdc565e4c824ee870d26692f95647caa0aaa09884021b2dcfd25f2df4d72"}
Oct 31 18:27:18.454885 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:18.454468    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:18.571253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:18.571197    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6" podStartSLOduration=2.696977384 podStartE2EDuration="16.571183868s" totalImagesPullingTime="13.874206484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:18.570554086 +0000 UTC m=+384.717003949" watchObservedRunningTime="2026-10-31 18:27:18.571183868 +0000 UTC m=+384.717633744"
Oct 31 18:27:18.754698 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:18.754661    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:18.954825 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:18.954777    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:19.108290 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.108210    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:19.291036 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.291002    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:19.455426 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.455304    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:19.641919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.641878    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hf75l"]
Oct 31 18:27:19.645498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.645464    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l"
Oct 31 18:27:19.652166 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.652145    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbl6q\" (UniqueName: \"kubernetes.io/projected/30e22b48-ac4a-4c9c-8154-79b160d5c529-kube-api-access-zbl6q\") pod \"kserve-controller-manager-776cc66d47-hf75l\" (UID: \"30e22b48-ac4a-4c9c-8154-79b160d5c529\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l"
Oct 31 18:27:19.652236 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.652209    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30e22b48-ac4a-4c9c-8154-79b160d5c529-cert\") pod \"kserve-controller-manager-776cc66d47-hf75l\" (UID: \"30e22b48-ac4a-4c9c-8154-79b160d5c529\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l"
Oct 31 18:27:19.710878 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.710772    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-w4t96\""
Oct 31 18:27:19.711184 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.711158    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 31 18:27:19.712272 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.712252    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hf75l"]
Oct 31 18:27:19.731402 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.731362    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:19.752686 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.752664    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30e22b48-ac4a-4c9c-8154-79b160d5c529-cert\") pod \"kserve-controller-manager-776cc66d47-hf75l\" (UID: \"30e22b48-ac4a-4c9c-8154-79b160d5c529\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l"
Oct 31 18:27:19.752840 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.752717    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbl6q\" (UniqueName: \"kubernetes.io/projected/30e22b48-ac4a-4c9c-8154-79b160d5c529-kube-api-access-zbl6q\") pod \"kserve-controller-manager-776cc66d47-hf75l\" (UID: \"30e22b48-ac4a-4c9c-8154-79b160d5c529\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l"
Oct 31 18:27:19.755327 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.755304    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30e22b48-ac4a-4c9c-8154-79b160d5c529-cert\") pod \"kserve-controller-manager-776cc66d47-hf75l\" (UID: \"30e22b48-ac4a-4c9c-8154-79b160d5c529\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l"
Oct 31 18:27:19.814886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.814857    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbl6q\" (UniqueName: \"kubernetes.io/projected/30e22b48-ac4a-4c9c-8154-79b160d5c529-kube-api-access-zbl6q\") pod \"kserve-controller-manager-776cc66d47-hf75l\" (UID: \"30e22b48-ac4a-4c9c-8154-79b160d5c529\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l"
Oct 31 18:27:19.840831 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.840798    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vhjgq"]
Oct 31 18:27:19.844244 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.844228    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq"
Oct 31 18:27:19.853685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.853663    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8-tls-certs\") pod \"model-serving-api-75bd89fdfc-vhjgq\" (UID: \"26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8\") " pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq"
Oct 31 18:27:19.853795 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.853707    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldfvg\" (UniqueName: \"kubernetes.io/projected/26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8-kube-api-access-ldfvg\") pod \"model-serving-api-75bd89fdfc-vhjgq\" (UID: \"26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8\") " pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq"
Oct 31 18:27:19.902839 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.902800    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vhjgq"]
Oct 31 18:27:19.910815 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.910789    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 31 18:27:19.910815 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.910810    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-sgll7\""
Oct 31 18:27:19.910981 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.910878    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:19.954596 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.954567    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8-tls-certs\") pod \"model-serving-api-75bd89fdfc-vhjgq\" (UID: \"26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8\") " pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq"
Oct 31 18:27:19.954734 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.954616    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldfvg\" (UniqueName: \"kubernetes.io/projected/26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8-kube-api-access-ldfvg\") pod \"model-serving-api-75bd89fdfc-vhjgq\" (UID: \"26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8\") " pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq"
Oct 31 18:27:19.957146 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.957121    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8-tls-certs\") pod \"model-serving-api-75bd89fdfc-vhjgq\" (UID: \"26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8\") " pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq"
Oct 31 18:27:19.957268 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:19.957223    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l"
Oct 31 18:27:20.013009 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:20.012984    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldfvg\" (UniqueName: \"kubernetes.io/projected/26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8-kube-api-access-ldfvg\") pod \"model-serving-api-75bd89fdfc-vhjgq\" (UID: \"26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8\") " pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq"
Oct 31 18:27:20.156671 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:20.156634    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq"
Oct 31 18:27:20.332739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:20.332712    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hf75l"]
Oct 31 18:27:20.335044 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:27:20.335016    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e22b48_ac4a_4c9c_8154_79b160d5c529.slice/crio-623a98bbdb825dae72b37c1f26a9d9a21736856090f7d7d21e3a0233502f36ce WatchSource:0}: Error finding container 623a98bbdb825dae72b37c1f26a9d9a21736856090f7d7d21e3a0233502f36ce: Status 404 returned error can't find the container with id 623a98bbdb825dae72b37c1f26a9d9a21736856090f7d7d21e3a0233502f36ce
Oct 31 18:27:20.463320 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:20.463281    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l" event={"ID":"30e22b48-ac4a-4c9c-8154-79b160d5c529","Type":"ContainerStarted","Data":"623a98bbdb825dae72b37c1f26a9d9a21736856090f7d7d21e3a0233502f36ce"}
Oct 31 18:27:20.529210 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:20.529188    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-vhjgq"]
Oct 31 18:27:20.531724 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:27:20.531690    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ab5dc0_00d0_4ce9_a2ef_207ad132bfb8.slice/crio-f090694da3aa8c75772c6e6c8728a2932eac97191ea8c3eaf04d733cc73c8e3a WatchSource:0}: Error finding container f090694da3aa8c75772c6e6c8728a2932eac97191ea8c3eaf04d733cc73c8e3a: Status 404 returned error can't find the container with id f090694da3aa8c75772c6e6c8728a2932eac97191ea8c3eaf04d733cc73c8e3a
Oct 31 18:27:21.043129 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:21.043083    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:21.469000 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:21.468927    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq" event={"ID":"26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8","Type":"ContainerStarted","Data":"f090694da3aa8c75772c6e6c8728a2932eac97191ea8c3eaf04d733cc73c8e3a"}
Oct 31 18:27:23.479319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:23.479288    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq" event={"ID":"26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8","Type":"ContainerStarted","Data":"47dc42fae0ad8704f9bb22ed7cf35b63e95eea70ebc3694d4d9601903dc5df27"}
Oct 31 18:27:23.479767 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:23.479418    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq"
Oct 31 18:27:24.484850 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:24.484811    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l" event={"ID":"30e22b48-ac4a-4c9c-8154-79b160d5c529","Type":"ContainerStarted","Data":"690333d94466a417c7aab21970b55e6bbb3f0ebc50f8ce301e74ec7ab86e78f1"}
Oct 31 18:27:24.600010 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:24.599959    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l" podStartSLOduration=2.174629473 podStartE2EDuration="5.599944545s" totalImagesPullingTime="3.425315072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:24.598969171 +0000 UTC m=+390.745419035" watchObservedRunningTime="2026-10-31 18:27:24.599944545 +0000 UTC m=+390.746394409"
Oct 31 18:27:24.600184 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:24.600044    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq" podStartSLOduration=3.359148801 podStartE2EDuration="5.600038203s" totalImagesPullingTime="2.240889402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:27:23.597667166 +0000 UTC m=+389.744117030" watchObservedRunningTime="2026-10-31 18:27:24.600038203 +0000 UTC m=+390.746488066"
Oct 31 18:27:25.467806 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:27:25.467767    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:27:25.488334 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:25.488309    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l"
Oct 31 18:27:26.076955 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:26.076892    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:26.273922 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:26.273885    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:26.425187 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:26.425093    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:26.582051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:26.582015    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:26.755396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:26.755364    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:26.926687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:26.926628    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:27.082459 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:27.082351    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:27.246151 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:27.246102    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:29.463950 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:29.463919    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-6xxg6"
Oct 31 18:27:33.175300 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:33.175263    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:33.392875 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:33.392837    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:34.488773 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:34.488747    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-vhjgq"
Oct 31 18:27:35.224875 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:35.224838    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:35.385179 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:35.385136    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:35.540105 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:35.540076    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:35.700601 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:35.700513    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:41.619588 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:41.619552    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:27:42.450781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:42.450738    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:49.696416 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:49.696373    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:49.904375 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:49.904333    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:27:55.467297 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:27:55.467256    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:27:56.497697 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:27:56.497671    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hf75l"
Oct 31 18:28:25.466940 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:28:25.466901    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:28:32.038565 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:28:32.038519    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:28:32.192228 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:28:32.192197    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:28:38.913621 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:28:38.913588    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:28:42.692415 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:28:42.692370    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:28:51.833378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:28:51.833336    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:28:55.467096 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:28:55.467058    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:29:09.673663 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.673580    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-69t2h"]
Oct 31 18:29:09.676745 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.676730    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h"
Oct 31 18:29:09.688188 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.688158    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:29:09.767289 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.767257    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9e97c75-1ba5-4d65-8636-669baf5d4032-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-69t2h\" (UID: \"a9e97c75-1ba5-4d65-8636-669baf5d4032\") " pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h"
Oct 31 18:29:09.767448 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.767309    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt7wn\" (UniqueName: \"kubernetes.io/projected/a9e97c75-1ba5-4d65-8636-669baf5d4032-kube-api-access-qt7wn\") pod \"ai-gateway-controller-7d85675f98-69t2h\" (UID: \"a9e97c75-1ba5-4d65-8636-669baf5d4032\") " pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h"
Oct 31 18:29:09.781875 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.781847    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 18:29:09.782036 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.782017    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 18:29:09.782150 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.782135    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-rb572\""
Oct 31 18:29:09.783316 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.783300    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-69t2h"]
Oct 31 18:29:09.865851 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.865819    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:29:09.867847 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.867824    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9e97c75-1ba5-4d65-8636-669baf5d4032-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-69t2h\" (UID: \"a9e97c75-1ba5-4d65-8636-669baf5d4032\") " pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h"
Oct 31 18:29:09.867956 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.867882    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt7wn\" (UniqueName: \"kubernetes.io/projected/a9e97c75-1ba5-4d65-8636-669baf5d4032-kube-api-access-qt7wn\") pod \"ai-gateway-controller-7d85675f98-69t2h\" (UID: \"a9e97c75-1ba5-4d65-8636-669baf5d4032\") " pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h"
Oct 31 18:29:09.870349 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.870325    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9e97c75-1ba5-4d65-8636-669baf5d4032-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-69t2h\" (UID: \"a9e97c75-1ba5-4d65-8636-669baf5d4032\") " pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h"
Oct 31 18:29:09.925833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.925757    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt7wn\" (UniqueName: \"kubernetes.io/projected/a9e97c75-1ba5-4d65-8636-669baf5d4032-kube-api-access-qt7wn\") pod \"ai-gateway-controller-7d85675f98-69t2h\" (UID: \"a9e97c75-1ba5-4d65-8636-669baf5d4032\") " pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h"
Oct 31 18:29:09.991499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:09.990101    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h"
Oct 31 18:29:10.355331 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:10.355308    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-69t2h"]
Oct 31 18:29:10.357457 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:29:10.357427    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e97c75_1ba5_4d65_8636_669baf5d4032.slice/crio-20dd4a749a428f2b0527a2f370366316d5377c7bd1dfb49b399a23f9cb268fdd WatchSource:0}: Error finding container 20dd4a749a428f2b0527a2f370366316d5377c7bd1dfb49b399a23f9cb268fdd: Status 404 returned error can't find the container with id 20dd4a749a428f2b0527a2f370366316d5377c7bd1dfb49b399a23f9cb268fdd
Oct 31 18:29:10.910157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:10.910120    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h" event={"ID":"a9e97c75-1ba5-4d65-8636-669baf5d4032","Type":"ContainerStarted","Data":"20dd4a749a428f2b0527a2f370366316d5377c7bd1dfb49b399a23f9cb268fdd"}
Oct 31 18:29:15.933217 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:15.933187    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h" event={"ID":"a9e97c75-1ba5-4d65-8636-669baf5d4032","Type":"ContainerStarted","Data":"a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb"}
Oct 31 18:29:15.933637 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:15.933299    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h"
Oct 31 18:29:16.047984 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:16.047929    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h" podStartSLOduration=2.510181447 podStartE2EDuration="7.04791735s" totalImagesPullingTime="4.537735903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:29:16.047272375 +0000 UTC m=+502.193722253" watchObservedRunningTime="2026-10-31 18:29:16.04791735 +0000 UTC m=+502.194367231"
Oct 31 18:29:17.579565 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:17.579535    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:29:17.786079 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:17.786039    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:29:17.962597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:17.962525    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"]
Oct 31 18:29:17.965873 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:17.965853    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"
Oct 31 18:29:18.032506 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.032456    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj627\" (UniqueName: \"kubernetes.io/projected/93867b3f-070c-4791-b3a9-55431f5cd3a4-kube-api-access-kj627\") pod \"payload-pre-processing-7b98bb678c-6ksnm\" (UID: \"93867b3f-070c-4791-b3a9-55431f5cd3a4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"
Oct 31 18:29:18.032662 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.032594    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93867b3f-070c-4791-b3a9-55431f5cd3a4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6ksnm\" (UID: \"93867b3f-070c-4791-b3a9-55431f5cd3a4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"
Oct 31 18:29:18.071023 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.070995    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 18:29:18.071157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.071140    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-p7jk2\""
Oct 31 18:29:18.072441 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.072416    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"]
Oct 31 18:29:18.114858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.114837    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-62pcj"]
Oct 31 18:29:18.118175 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.118159    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj"
Oct 31 18:29:18.133156 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.133128    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44a1311-1de6-46c3-8070-07fa09d89854-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-62pcj\" (UID: \"a44a1311-1de6-46c3-8070-07fa09d89854\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj"
Oct 31 18:29:18.133260 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.133164    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93867b3f-070c-4791-b3a9-55431f5cd3a4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6ksnm\" (UID: \"93867b3f-070c-4791-b3a9-55431f5cd3a4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"
Oct 31 18:29:18.133305 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.133277    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj627\" (UniqueName: \"kubernetes.io/projected/93867b3f-070c-4791-b3a9-55431f5cd3a4-kube-api-access-kj627\") pod \"payload-pre-processing-7b98bb678c-6ksnm\" (UID: \"93867b3f-070c-4791-b3a9-55431f5cd3a4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"
Oct 31 18:29:18.133350 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.133322    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llqgr\" (UniqueName: \"kubernetes.io/projected/a44a1311-1de6-46c3-8070-07fa09d89854-kube-api-access-llqgr\") pod \"payload-processing-55fbbbf9dc-62pcj\" (UID: \"a44a1311-1de6-46c3-8070-07fa09d89854\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj"
Oct 31 18:29:18.133849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.133831    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93867b3f-070c-4791-b3a9-55431f5cd3a4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6ksnm\" (UID: \"93867b3f-070c-4791-b3a9-55431f5cd3a4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"
Oct 31 18:29:18.176382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.176358    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-62pcj"]
Oct 31 18:29:18.190560 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.190537    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj627\" (UniqueName: \"kubernetes.io/projected/93867b3f-070c-4791-b3a9-55431f5cd3a4-kube-api-access-kj627\") pod \"payload-pre-processing-7b98bb678c-6ksnm\" (UID: \"93867b3f-070c-4791-b3a9-55431f5cd3a4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"
Oct 31 18:29:18.234426 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.234372    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44a1311-1de6-46c3-8070-07fa09d89854-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-62pcj\" (UID: \"a44a1311-1de6-46c3-8070-07fa09d89854\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj"
Oct 31 18:29:18.234561 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.234436    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llqgr\" (UniqueName: \"kubernetes.io/projected/a44a1311-1de6-46c3-8070-07fa09d89854-kube-api-access-llqgr\") pod \"payload-processing-55fbbbf9dc-62pcj\" (UID: \"a44a1311-1de6-46c3-8070-07fa09d89854\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj"
Oct 31 18:29:18.235291 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.235266    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44a1311-1de6-46c3-8070-07fa09d89854-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-62pcj\" (UID: \"a44a1311-1de6-46c3-8070-07fa09d89854\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj"
Oct 31 18:29:18.277984 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.277959    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"
Oct 31 18:29:18.291588 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.291561    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llqgr\" (UniqueName: \"kubernetes.io/projected/a44a1311-1de6-46c3-8070-07fa09d89854-kube-api-access-llqgr\") pod \"payload-processing-55fbbbf9dc-62pcj\" (UID: \"a44a1311-1de6-46c3-8070-07fa09d89854\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj"
Oct 31 18:29:18.430114 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.430084    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj"
Oct 31 18:29:18.644742 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.644718    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"]
Oct 31 18:29:18.647174 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:29:18.647136    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93867b3f_070c_4791_b3a9_55431f5cd3a4.slice/crio-859a8e38c37cf28b83298ea9e80d27321619c54506b8e35152de232f5f9cca0c WatchSource:0}: Error finding container 859a8e38c37cf28b83298ea9e80d27321619c54506b8e35152de232f5f9cca0c: Status 404 returned error can't find the container with id 859a8e38c37cf28b83298ea9e80d27321619c54506b8e35152de232f5f9cca0c
Oct 31 18:29:18.798073 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.798051    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-62pcj"]
Oct 31 18:29:18.799300 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:29:18.799270    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44a1311_1de6_46c3_8070_07fa09d89854.slice/crio-228487c8bf954bad71d099d766f28b4031c0b9f3b74c8b8a0f52d032f8f6533d WatchSource:0}: Error finding container 228487c8bf954bad71d099d766f28b4031c0b9f3b74c8b8a0f52d032f8f6533d: Status 404 returned error can't find the container with id 228487c8bf954bad71d099d766f28b4031c0b9f3b74c8b8a0f52d032f8f6533d
Oct 31 18:29:18.946686 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.946599    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm" event={"ID":"93867b3f-070c-4791-b3a9-55431f5cd3a4","Type":"ContainerStarted","Data":"859a8e38c37cf28b83298ea9e80d27321619c54506b8e35152de232f5f9cca0c"}
Oct 31 18:29:18.947794 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:18.947773    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj" event={"ID":"a44a1311-1de6-46c3-8070-07fa09d89854","Type":"ContainerStarted","Data":"228487c8bf954bad71d099d766f28b4031c0b9f3b74c8b8a0f52d032f8f6533d"}
Oct 31 18:29:21.963702 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:21.963656    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm" event={"ID":"93867b3f-070c-4791-b3a9-55431f5cd3a4","Type":"ContainerStarted","Data":"8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a"}
Oct 31 18:29:21.964181 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:21.963764    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"
Oct 31 18:29:21.965188 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:21.965165    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj" event={"ID":"a44a1311-1de6-46c3-8070-07fa09d89854","Type":"ContainerStarted","Data":"3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f"}
Oct 31 18:29:21.965328 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:21.965309    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj"
Oct 31 18:29:22.078671 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:22.078617    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm" podStartSLOduration=2.228053217 podStartE2EDuration="5.07860227s" totalImagesPullingTime="2.850549053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:29:22.077670825 +0000 UTC m=+508.224120691" watchObservedRunningTime="2026-10-31 18:29:22.07860227 +0000 UTC m=+508.225052131"
Oct 31 18:29:22.191722 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:22.191677    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj" podStartSLOduration=1.525076163 podStartE2EDuration="4.191663866s" totalImagesPullingTime="2.666587703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:29:22.190848566 +0000 UTC m=+508.337298433" watchObservedRunningTime="2026-10-31 18:29:22.191663866 +0000 UTC m=+508.338113760"
Oct 31 18:29:25.466567 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:29:25.466529    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:29:26.939907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:26.939874    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h"
Oct 31 18:29:27.324310 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:27.324277    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"]
Oct 31 18:29:27.327806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:27.327791    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"
Oct 31 18:29:27.384811 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:27.384782    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"]
Oct 31 18:29:27.409071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:27.409041    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqlb2\" (UniqueName: \"kubernetes.io/projected/ea33ee05-e391-4d64-a57a-039b32d83cd2-kube-api-access-vqlb2\") pod \"ai-gateway-controller-c8b7d8cc9-rzk2k\" (UID: \"ea33ee05-e391-4d64-a57a-039b32d83cd2\") " pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"
Oct 31 18:29:27.409222 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:27.409166    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea33ee05-e391-4d64-a57a-039b32d83cd2-metrics-certs\") pod \"ai-gateway-controller-c8b7d8cc9-rzk2k\" (UID: \"ea33ee05-e391-4d64-a57a-039b32d83cd2\") " pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"
Oct 31 18:29:27.509886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:27.509850    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea33ee05-e391-4d64-a57a-039b32d83cd2-metrics-certs\") pod \"ai-gateway-controller-c8b7d8cc9-rzk2k\" (UID: \"ea33ee05-e391-4d64-a57a-039b32d83cd2\") " pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"
Oct 31 18:29:27.510032 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:27.509897    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqlb2\" (UniqueName: \"kubernetes.io/projected/ea33ee05-e391-4d64-a57a-039b32d83cd2-kube-api-access-vqlb2\") pod \"ai-gateway-controller-c8b7d8cc9-rzk2k\" (UID: \"ea33ee05-e391-4d64-a57a-039b32d83cd2\") " pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"
Oct 31 18:29:27.512364 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:27.512338    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea33ee05-e391-4d64-a57a-039b32d83cd2-metrics-certs\") pod \"ai-gateway-controller-c8b7d8cc9-rzk2k\" (UID: \"ea33ee05-e391-4d64-a57a-039b32d83cd2\") " pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"
Oct 31 18:29:27.568850 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:27.568811    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqlb2\" (UniqueName: \"kubernetes.io/projected/ea33ee05-e391-4d64-a57a-039b32d83cd2-kube-api-access-vqlb2\") pod \"ai-gateway-controller-c8b7d8cc9-rzk2k\" (UID: \"ea33ee05-e391-4d64-a57a-039b32d83cd2\") " pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"
Oct 31 18:29:27.639522 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:27.639420    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"
Oct 31 18:29:28.005303 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:28.005283    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"]
Oct 31 18:29:28.007724 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:29:28.007699    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea33ee05_e391_4d64_a57a_039b32d83cd2.slice/crio-cca81ab7ccd153c9ac19d6ae8a5a20049dae1207c5c291965a890ec805258592 WatchSource:0}: Error finding container cca81ab7ccd153c9ac19d6ae8a5a20049dae1207c5c291965a890ec805258592: Status 404 returned error can't find the container with id cca81ab7ccd153c9ac19d6ae8a5a20049dae1207c5c291965a890ec805258592
Oct 31 18:29:28.995017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:28.994978    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k" event={"ID":"ea33ee05-e391-4d64-a57a-039b32d83cd2","Type":"ContainerStarted","Data":"1d38493a99a1736c5921b2e4d3800a1c02eb763fb9added986643df1dbb9ea9b"}
Oct 31 18:29:28.995184 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:28.995031    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k" event={"ID":"ea33ee05-e391-4d64-a57a-039b32d83cd2","Type":"ContainerStarted","Data":"cca81ab7ccd153c9ac19d6ae8a5a20049dae1207c5c291965a890ec805258592"}
Oct 31 18:29:28.995184 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:28.995113    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"
Oct 31 18:29:29.110169 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:29.110120    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k" podStartSLOduration=1.2166441080000001 podStartE2EDuration="2.11010503s" totalImagesPullingTime="893.460922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:29:29.109355333 +0000 UTC m=+515.255805198" watchObservedRunningTime="2026-10-31 18:29:29.11010503 +0000 UTC m=+515.256554893"
Oct 31 18:29:32.972767 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:32.972739    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"
Oct 31 18:29:32.973137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:32.972805    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj"
Oct 31 18:29:40.001908 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:40.001882    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"
Oct 31 18:29:40.168396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:40.168363    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-69t2h"]
Oct 31 18:29:40.168660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:40.168616    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h" podUID="a9e97c75-1ba5-4d65-8636-669baf5d4032" containerName="manager" containerID="cri-o://a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb" gracePeriod=30
Oct 31 18:29:40.415488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:40.415446    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h"
Oct 31 18:29:40.512442 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:40.512409    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e97c75-1ba5-4d65-8636-669baf5d4032-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9e97c75-1ba5-4d65-8636-669baf5d4032-metrics-certs\") pod \"a9e97c75-1ba5-4d65-8636-669baf5d4032\" (UID: \"a9e97c75-1ba5-4d65-8636-669baf5d4032\") "
Oct 31 18:29:40.512637 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:40.512513    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e97c75-1ba5-4d65-8636-669baf5d4032-kube-api-access-qt7wn\" (UniqueName: \"kubernetes.io/projected/a9e97c75-1ba5-4d65-8636-669baf5d4032-kube-api-access-qt7wn\") pod \"a9e97c75-1ba5-4d65-8636-669baf5d4032\" (UID: \"a9e97c75-1ba5-4d65-8636-669baf5d4032\") "
Oct 31 18:29:40.514782 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:40.514761    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e97c75-1ba5-4d65-8636-669baf5d4032-metrics-certs" pod "a9e97c75-1ba5-4d65-8636-669baf5d4032" (UID: "a9e97c75-1ba5-4d65-8636-669baf5d4032"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:29:40.514862 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:40.514761    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e97c75-1ba5-4d65-8636-669baf5d4032-kube-api-access-qt7wn" pod "a9e97c75-1ba5-4d65-8636-669baf5d4032" (UID: "a9e97c75-1ba5-4d65-8636-669baf5d4032"). InnerVolumeSpecName "kube-api-access-qt7wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:29:40.613788 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:40.613704    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9e97c75-1ba5-4d65-8636-669baf5d4032-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:29:40.613788 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:40.613742    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt7wn\" (UniqueName: \"kubernetes.io/projected/a9e97c75-1ba5-4d65-8636-669baf5d4032-kube-api-access-qt7wn\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:29:41.047951 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:41.047914    4484 generic.go:356] "Generic (PLEG): container finished" podID="a9e97c75-1ba5-4d65-8636-669baf5d4032" containerID="a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb" exitCode=0
Oct 31 18:29:41.048507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:41.047971    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h"
Oct 31 18:29:41.048507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:41.047977    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h" event={"ID":"a9e97c75-1ba5-4d65-8636-669baf5d4032","Type":"ContainerDied","Data":"a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb"}
Oct 31 18:29:41.048507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:41.048029    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-69t2h" event={"ID":"a9e97c75-1ba5-4d65-8636-669baf5d4032","Type":"ContainerDied","Data":"20dd4a749a428f2b0527a2f370366316d5377c7bd1dfb49b399a23f9cb268fdd"}
Oct 31 18:29:41.048507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:41.048060    4484 scope.go:122] "RemoveContainer" containerID="a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb"
Oct 31 18:29:41.057251 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:41.057230    4484 scope.go:122] "RemoveContainer" containerID="a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb"
Oct 31 18:29:41.057555 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:29:41.057537    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb\": container with ID starting with a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb not found: ID does not exist" containerID="a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb"
Oct 31 18:29:41.057615 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:41.057567    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb"} err="failed to get container status \"a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb\": rpc error: code = NotFound desc = could not find container \"a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb\": container with ID starting with a2587a89e37da6c633875677084c48bec2d659b84265f25b51a5a5e6e3af3aeb not found: ID does not exist"
Oct 31 18:29:41.215831 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:41.215742    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-69t2h"]
Oct 31 18:29:41.217549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:41.217527    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-69t2h"]
Oct 31 18:29:41.792017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:41.791983    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:29:41.991313 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:41.991272    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e97c75-1ba5-4d65-8636-669baf5d4032" path="/var/lib/kubelet/pods/a9e97c75-1ba5-4d65-8636-669baf5d4032/volumes"
Oct 31 18:29:42.290467 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:42.290432    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:29:43.044045 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:43.044012    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:29:43.530155 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:43.530124    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:29:43.689015 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:43.688982    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:29:50.045822 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.045786    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"]
Oct 31 18:29:50.046301 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.046281    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a9e97c75-1ba5-4d65-8636-669baf5d4032" containerName="manager"
Oct 31 18:29:50.046376 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.046306    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e97c75-1ba5-4d65-8636-669baf5d4032" containerName="manager"
Oct 31 18:29:50.046430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.046418    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e97c75-1ba5-4d65-8636-669baf5d4032" containerName="manager"
Oct 31 18:29:50.052638 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.052613    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.064277 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.064254    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:29:50.094591 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.094562    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.094749 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.094638    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.094749 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.094702    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfn6l\" (UniqueName: \"kubernetes.io/projected/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-kube-api-access-lfn6l\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.094837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.094811    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.094872 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.094858    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.094950 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.094931    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.132655 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.132631    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v7tpx\""
Oct 31 18:29:50.155825 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.155796    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 18:29:50.155825 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.155796    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 18:29:50.155996 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.155837    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 31 18:29:50.157286 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.157265    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"]
Oct 31 18:29:50.195664 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.195634    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfn6l\" (UniqueName: \"kubernetes.io/projected/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-kube-api-access-lfn6l\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.195807 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.195689    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.195807 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.195721    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.195807 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.195748    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.195807 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.195787    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.195981 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.195811    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.196118 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.196099    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.196188 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.196168    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.196243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.196174    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.198083 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.198061    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.198406 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.198387    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.227254 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.227222    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:29:50.254328 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.254299    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfn6l\" (UniqueName: \"kubernetes.io/projected/e86af09b-1972-4b3a-9c2e-7ba2b8b83670-kube-api-access-lfn6l\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj\" (UID: \"e86af09b-1972-4b3a-9c2e-7ba2b8b83670\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.365266 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.365169    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:50.741975 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:50.741956    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"]
Oct 31 18:29:50.744033 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:29:50.744009    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86af09b_1972_4b3a_9c2e_7ba2b8b83670.slice/crio-70e81544627e9946a2007648f0eb4e02008a592d556b71022739bdda78bce330 WatchSource:0}: Error finding container 70e81544627e9946a2007648f0eb4e02008a592d556b71022739bdda78bce330: Status 404 returned error can't find the container with id 70e81544627e9946a2007648f0eb4e02008a592d556b71022739bdda78bce330
Oct 31 18:29:51.089955 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:51.089866    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj" event={"ID":"e86af09b-1972-4b3a-9c2e-7ba2b8b83670","Type":"ContainerStarted","Data":"70e81544627e9946a2007648f0eb4e02008a592d556b71022739bdda78bce330"}
Oct 31 18:29:53.099642 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:53.099610    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj" event={"ID":"e86af09b-1972-4b3a-9c2e-7ba2b8b83670","Type":"ContainerStarted","Data":"bc907fc861122e5e3fd9e9341c37b59f35d8c73031f4360a57a3db06b64ec20c"}
Oct 31 18:29:53.215780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:53.215722    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj" podStartSLOduration=0.989829927 podStartE2EDuration="3.215709485s" totalImagesPullingTime="2.225879558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:29:53.214820747 +0000 UTC m=+539.361270611" watchObservedRunningTime="2026-10-31 18:29:53.215709485 +0000 UTC m=+539.362159379"
Oct 31 18:29:53.365987 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:53.365954    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:29:53.818567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:53.818531    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:29:55.467845 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:29:55.467807    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:29:56.951571 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:56.951531    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"]
Oct 31 18:29:56.956537 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:56.956515    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.058955 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.058916    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6683cf79-6892-4279-b3ad-65a6229a4a18-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.059143 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.059034    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6683cf79-6892-4279-b3ad-65a6229a4a18-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.059143 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.059070    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6683cf79-6892-4279-b3ad-65a6229a4a18-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.059143 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.059088    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6683cf79-6892-4279-b3ad-65a6229a4a18-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.059256 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.059208    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6683cf79-6892-4279-b3ad-65a6229a4a18-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.059256 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.059236    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhfk\" (UniqueName: \"kubernetes.io/projected/6683cf79-6892-4279-b3ad-65a6229a4a18-kube-api-access-dwhfk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.060849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.060833    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 31 18:29:57.062504 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.062446    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:29:57.062613 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.062576    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"]
Oct 31 18:29:57.160214 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.160185    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6683cf79-6892-4279-b3ad-65a6229a4a18-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.160372 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.160242    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6683cf79-6892-4279-b3ad-65a6229a4a18-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.160372 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.160354    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6683cf79-6892-4279-b3ad-65a6229a4a18-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.160516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.160380    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6683cf79-6892-4279-b3ad-65a6229a4a18-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.160516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.160402    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6683cf79-6892-4279-b3ad-65a6229a4a18-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.160516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.160420    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhfk\" (UniqueName: \"kubernetes.io/projected/6683cf79-6892-4279-b3ad-65a6229a4a18-kube-api-access-dwhfk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.160752 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.160719    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6683cf79-6892-4279-b3ad-65a6229a4a18-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.160752 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.160748    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6683cf79-6892-4279-b3ad-65a6229a4a18-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.160900 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.160799    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6683cf79-6892-4279-b3ad-65a6229a4a18-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.162697 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.162676    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6683cf79-6892-4279-b3ad-65a6229a4a18-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.162848 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.162831    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6683cf79-6892-4279-b3ad-65a6229a4a18-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.217336 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.217290    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhfk\" (UniqueName: \"kubernetes.io/projected/6683cf79-6892-4279-b3ad-65a6229a4a18-kube-api-access-dwhfk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-lwj8x\" (UID: \"6683cf79-6892-4279-b3ad-65a6229a4a18\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.268440 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.268408    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:29:57.639407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:57.639379    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"]
Oct 31 18:29:57.641033 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:29:57.641007    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6683cf79_6892_4279_b3ad_65a6229a4a18.slice/crio-a3a75d6585e7ee4bb05e3a64b42bd7eaf5bc0284acd7d626c3ee8f45ce392b2e WatchSource:0}: Error finding container a3a75d6585e7ee4bb05e3a64b42bd7eaf5bc0284acd7d626c3ee8f45ce392b2e: Status 404 returned error can't find the container with id a3a75d6585e7ee4bb05e3a64b42bd7eaf5bc0284acd7d626c3ee8f45ce392b2e
Oct 31 18:29:58.120683 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:58.120650    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x" event={"ID":"6683cf79-6892-4279-b3ad-65a6229a4a18","Type":"ContainerStarted","Data":"a3a75d6585e7ee4bb05e3a64b42bd7eaf5bc0284acd7d626c3ee8f45ce392b2e"}
Oct 31 18:29:59.126262 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:59.126227    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x" event={"ID":"6683cf79-6892-4279-b3ad-65a6229a4a18","Type":"ContainerStarted","Data":"30e30d5689f7c682159efd0eb903f2274dfdbdfc9e1c18f1d7f6ac83a29cc517"}
Oct 31 18:29:59.242843 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:59.242792    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x" podStartSLOduration=2.669046539 podStartE2EDuration="3.242777702s" totalImagesPullingTime="573.731163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:29:59.241213928 +0000 UTC m=+545.387663791" watchObservedRunningTime="2026-10-31 18:29:59.242777702 +0000 UTC m=+545.389227564"
Oct 31 18:29:59.268722 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:29:59.268686    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:30:00.365859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:00.365825    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:30:00.378063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:00.378039    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:30:01.004293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:01.004259    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:30:01.145467 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:01.145437    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj"
Oct 31 18:30:04.526346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:04.526312    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:30:04.832822 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:04.832747    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"]
Oct 31 18:30:04.836324 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:04.836309    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:04.927747 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:04.927717    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:04.927747 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:04.927749    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:04.928000 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:04.927838    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:04.928000 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:04.927857    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:04.928000 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:04.927943    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j64lj\" (UniqueName: \"kubernetes.io/projected/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-kube-api-access-j64lj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:04.928123 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:04.927998    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:04.942031 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:04.942009    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 31 18:30:04.943499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:04.943467    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"]
Oct 31 18:30:04.945798 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:04.945773    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:30:05.029412 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.029379    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.029581 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.029435    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.029581 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.029467    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.029581 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.029559    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.029739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.029582    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.029739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.029620    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64lj\" (UniqueName: \"kubernetes.io/projected/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-kube-api-access-j64lj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.029941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.029910    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.030046 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.029920    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.030046 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.029920    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.031980 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.031952    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.032119 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.032104    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.091639 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.091581    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64lj\" (UniqueName: \"kubernetes.io/projected/7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2-kube-api-access-j64lj\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8k7kf\" (UID: \"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.147877 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.147853    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:05.514862 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:05.514839    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"]
Oct 31 18:30:05.516775 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:30:05.516754    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc3fdbe_f1d9_4d47_82d9_e46f3366d5d2.slice/crio-5bf8f3008e3cca3f5928a3cc145173945c1547eae52ab4958f832dcdf1536d04 WatchSource:0}: Error finding container 5bf8f3008e3cca3f5928a3cc145173945c1547eae52ab4958f832dcdf1536d04: Status 404 returned error can't find the container with id 5bf8f3008e3cca3f5928a3cc145173945c1547eae52ab4958f832dcdf1536d04
Oct 31 18:30:06.158651 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:06.158620    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf" event={"ID":"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2","Type":"ContainerStarted","Data":"5bf8f3008e3cca3f5928a3cc145173945c1547eae52ab4958f832dcdf1536d04"}
Oct 31 18:30:07.164607 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:07.164566    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf" event={"ID":"7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2","Type":"ContainerStarted","Data":"5ee94d60821960fe428bc09661d01d70431fba3ccc4e04adcf4ca921427e237d"}
Oct 31 18:30:07.268733 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:07.268699    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:30:07.280300 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:07.280249    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf" podStartSLOduration=2.680864087 podStartE2EDuration="3.280236645s" totalImagesPullingTime="599.372558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:30:07.279457181 +0000 UTC m=+553.425907058" watchObservedRunningTime="2026-10-31 18:30:07.280236645 +0000 UTC m=+553.426686508"
Oct 31 18:30:07.281412 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:07.281389    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:30:08.147988 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:08.147965    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:08.179545 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:08.179521    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-lwj8x"
Oct 31 18:30:08.643694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:08.643659    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:30:12.254632 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:12.254596    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:30:15.148449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:15.148413    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:15.160255 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:15.160234    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:15.209229 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:15.209203    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8k7kf"
Oct 31 18:30:15.785832 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:15.785798    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:30:16.018615 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.018591    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"]
Oct 31 18:30:16.022035 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.022020    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.076961 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.076882    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 31 18:30:16.128287 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.128257    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"]
Oct 31 18:30:16.130581 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.130562    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.130652 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.130590    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.130652 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.130607    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xclnl\" (UniqueName: \"kubernetes.io/projected/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-kube-api-access-xclnl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.130783 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.130765    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.130844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.130825    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.130951 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.130921    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.153366 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.153318    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:30:16.232336 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.232305    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.232560 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.232359    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.232560 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.232400    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.232560 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.232434    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.232560 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.232459    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.232560 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.232511    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xclnl\" (UniqueName: \"kubernetes.io/projected/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-kube-api-access-xclnl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.232814 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.232789    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.232862 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.232798    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.232862 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.232850    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.234574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.234555    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.234936 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.234918    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.296724 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.296698    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xclnl\" (UniqueName: \"kubernetes.io/projected/c1e45f78-79e3-4e67-a2c2-0ff4a291374b-kube-api-access-xclnl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd\" (UID: \"c1e45f78-79e3-4e67-a2c2-0ff4a291374b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.333606 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.333531    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:16.702685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:16.702661    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"]
Oct 31 18:30:16.704549 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:30:16.704522    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e45f78_79e3_4e67_a2c2_0ff4a291374b.slice/crio-f43f3cae199448aa52e950db5c7789da15d7881a83df7257501b695b8b731f03 WatchSource:0}: Error finding container f43f3cae199448aa52e950db5c7789da15d7881a83df7257501b695b8b731f03: Status 404 returned error can't find the container with id f43f3cae199448aa52e950db5c7789da15d7881a83df7257501b695b8b731f03
Oct 31 18:30:17.209624 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.209516    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd" event={"ID":"c1e45f78-79e3-4e67-a2c2-0ff4a291374b","Type":"ContainerStarted","Data":"f43f3cae199448aa52e950db5c7789da15d7881a83df7257501b695b8b731f03"}
Oct 31 18:30:17.219724 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.219698    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"]
Oct 31 18:30:17.224331 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.224315    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.242522 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.242498    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.242633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.242534    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.242633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.242560    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675a6c4e-5415-4741-b381-257fd4c918c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.242706 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.242640    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdpzn\" (UniqueName: \"kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-kube-api-access-bdpzn\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.242740 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.242702    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.242740 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.242723    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675a6c4e-5415-4741-b381-257fd4c918c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.242817 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.242741    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.242817 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.242766    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.242882 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.242826    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.277593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.277566    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 18:30:17.277765 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.277582    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nsw28\""
Oct 31 18:30:17.330489 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.330448    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"]
Oct 31 18:30:17.343501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.343457    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675a6c4e-5415-4741-b381-257fd4c918c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.343604 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.343534    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdpzn\" (UniqueName: \"kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-kube-api-access-bdpzn\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.343604 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.343566    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.343604 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.343592    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675a6c4e-5415-4741-b381-257fd4c918c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.343783 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.343734    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.343783 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.343778    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.343894 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.343826    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.343949 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.343902    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.343949 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.343938    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.344049 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.343962    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.344101 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.344083    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.344155 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.344109    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.344155 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.344108    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675a6c4e-5415-4741-b381-257fd4c918c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.344241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.344220    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.346585 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.346562    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.346887 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.346862    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675a6c4e-5415-4741-b381-257fd4c918c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.400708 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.400679    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdpzn\" (UniqueName: \"kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-kube-api-access-bdpzn\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.400818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.400790    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4xgvt\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.537534 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.537498    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:17.958681 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:17.958654    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"]
Oct 31 18:30:17.960503 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:30:17.960461    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675a6c4e_5415_4741_b381_257fd4c918c1.slice/crio-0565c0bab6d4256298673adfa7c5639b8f654e514d321e323919e972b198c4d1 WatchSource:0}: Error finding container 0565c0bab6d4256298673adfa7c5639b8f654e514d321e323919e972b198c4d1: Status 404 returned error can't find the container with id 0565c0bab6d4256298673adfa7c5639b8f654e514d321e323919e972b198c4d1
Oct 31 18:30:18.216003 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:18.215914    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd" event={"ID":"c1e45f78-79e3-4e67-a2c2-0ff4a291374b","Type":"ContainerStarted","Data":"703f7cd790ffdd09635cc100b41a22151186aa79f67785605e5b20c81084300c"}
Oct 31 18:30:18.217334 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:18.217308    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" event={"ID":"675a6c4e-5415-4741-b381-257fd4c918c1","Type":"ContainerStarted","Data":"0565c0bab6d4256298673adfa7c5639b8f654e514d321e323919e972b198c4d1"}
Oct 31 18:30:18.331418 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:18.331372    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd" podStartSLOduration=2.728375406 podStartE2EDuration="3.33136003s" totalImagesPullingTime="602.984624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:30:18.330162844 +0000 UTC m=+564.476612708" watchObservedRunningTime="2026-10-31 18:30:18.33136003 +0000 UTC m=+564.477809893"
Oct 31 18:30:18.334209 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:18.334188    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:19.780615 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:19.780582    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:30:21.814972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:21.814934    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:30:21.815338 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:21.815015    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:30:21.815338 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:21.815063    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:30:22.247988 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:22.247952    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" event={"ID":"675a6c4e-5415-4741-b381-257fd4c918c1","Type":"ContainerStarted","Data":"1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67"}
Oct 31 18:30:22.416241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:22.416188    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" podStartSLOduration=1.564609521 podStartE2EDuration="5.416174417s" totalImagesPullingTime="3.851564896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:30:22.41461593 +0000 UTC m=+568.561065805" watchObservedRunningTime="2026-10-31 18:30:22.416174417 +0000 UTC m=+568.562624279"
Oct 31 18:30:22.538096 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:22.538012    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:22.540082 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:22.539451    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4xgvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 18:30:22.540082 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:22.539525    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 18:30:23.537859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:23.537828    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4xgvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 18:30:23.538255 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:23.537894    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 18:30:24.538583 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:24.538553    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4xgvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 18:30:24.538972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:24.538606    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 18:30:25.495346 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:30:25.495307    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:30:25.538319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:25.538286    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4xgvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 18:30:25.538496 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:25.538340    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 18:30:26.334505 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:26.334458    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:26.346763 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:26.346730    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:26.538127 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:26.538091    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4xgvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 18:30:26.538294 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:26.538138    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 18:30:27.304483 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:27.304455    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd"
Oct 31 18:30:27.538270 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:27.538236    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4xgvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 18:30:27.538674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:27.538303    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 18:30:28.538640 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:28.538597    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4xgvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 18:30:28.539161 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:28.538652    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 18:30:29.951848 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:29.951805    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:30:30.579659 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:30.579615    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4xgvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:30:30.579840 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:30.579704    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:30:31.190883 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:31.190851    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:31.191497 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:31.191172    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:31.191997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:31.191977    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:30:43.363872 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:43.363793    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:30:46.592398 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:46.592362    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:30:46.755997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:46.755963    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:30:47.465908 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.465872    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"]
Oct 31 18:30:47.469932 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.469913    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.574757 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.574724    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"]
Oct 31 18:30:47.618746 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.618716    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.618746 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.618746    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkqr5\" (UniqueName: \"kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-kube-api-access-dkqr5\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.619225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.618934    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.619225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.618972    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.619225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.619056    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.619225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.619073    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.619225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.619089    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68c29f1-193e-40f4-bcb5-62e42a865c98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.619225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.619106    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.619225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.619121    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.719851 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.719783    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.719851 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.719816    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.719851 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.719833    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68c29f1-193e-40f4-bcb5-62e42a865c98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.719851 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.719850    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.720151 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.719864    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.720151 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.719891    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.720151 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.719913    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqr5\" (UniqueName: \"kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-kube-api-access-dkqr5\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.720151 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.720009    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.720151 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.720035    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.720417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.720212    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.720566 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.720543    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.720638 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.720592    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.720638 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.720622    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.720903 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.720885    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68c29f1-193e-40f4-bcb5-62e42a865c98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.722864 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.722843    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.722994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.722974    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.778766 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.778734    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.778898 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.778774    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqr5\" (UniqueName: \"kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-kube-api-access-dkqr5\") pod \"maas-default-gateway-openshift-default-7f694df8bc-ktlzz\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:47.782604 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:47.782586    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:48.203225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:48.203200    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"]
Oct 31 18:30:48.204814 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:30:48.204787    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68c29f1_193e_40f4_bcb5_62e42a865c98.slice/crio-02b081022f61db44e953c4aef9049badd3fcc0c3bf49f3533c8abc60fd79f50a WatchSource:0}: Error finding container 02b081022f61db44e953c4aef9049badd3fcc0c3bf49f3533c8abc60fd79f50a: Status 404 returned error can't find the container with id 02b081022f61db44e953c4aef9049badd3fcc0c3bf49f3533c8abc60fd79f50a
Oct 31 18:30:48.207105 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:48.207074    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:30:48.207179 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:48.207143    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:30:48.207179 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:48.207170    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:30:48.238346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:48.238301    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:30:48.392863 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:48.392824    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz" event={"ID":"e68c29f1-193e-40f4-bcb5-62e42a865c98","Type":"ContainerStarted","Data":"1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b"}
Oct 31 18:30:48.393040 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:48.392913    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz" event={"ID":"e68c29f1-193e-40f4-bcb5-62e42a865c98","Type":"ContainerStarted","Data":"02b081022f61db44e953c4aef9049badd3fcc0c3bf49f3533c8abc60fd79f50a"}
Oct 31 18:30:48.401785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:48.401759    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:30:48.511980 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:48.511930    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz" podStartSLOduration=1.5119178039999999 podStartE2EDuration="1.511917804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:30:48.511748653 +0000 UTC m=+594.658198517" watchObservedRunningTime="2026-10-31 18:30:48.511917804 +0000 UTC m=+594.658367667"
Oct 31 18:30:48.785672 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:48.785574    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:49.785500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:49.785246    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ktlzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 18:30:49.785500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:49.785317    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz" podUID="e68c29f1-193e-40f4-bcb5-62e42a865c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 18:30:50.785499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:50.784968    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ktlzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 18:30:50.785499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:50.785045    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz" podUID="e68c29f1-193e-40f4-bcb5-62e42a865c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 18:30:51.785954 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:51.785921    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ktlzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 18:30:51.786457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:51.785972    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz" podUID="e68c29f1-193e-40f4-bcb5-62e42a865c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 18:30:52.785917 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:52.785877    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ktlzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 18:30:52.786092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:52.785947    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz" podUID="e68c29f1-193e-40f4-bcb5-62e42a865c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 18:30:53.785875 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:53.785836    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ktlzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 18:30:53.786059 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:53.785892    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz" podUID="e68c29f1-193e-40f4-bcb5-62e42a865c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 18:30:54.042643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:54.042555    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:30:54.046110 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:54.046092    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:30:54.783890 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:54.783856    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-ktlzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 18:30:54.784064 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:54.783908    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz" podUID="e68c29f1-193e-40f4-bcb5-62e42a865c98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 18:30:55.467333 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:30:55.467300    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:30:56.246262 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:56.246231    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:56.246785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:56.246726    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:30:56.247490 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:30:56.247457    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 18:31:16.686145 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:16.686111    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:16.990605 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:16.990571    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:17.252031 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.251958    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:17.464433 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.464398    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:17.625219 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.625148    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"]
Oct 31 18:31:17.628583 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.628567    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.630912 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.630884    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:17.697384 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.697351    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-certs\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.697384 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.697387    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lznrc\" (UniqueName: \"kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-kube-api-access-lznrc\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.697899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.697405    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-podinfo\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.697899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.697438    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-envoy\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.697899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.697555    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-socket\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.697899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.697576    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-credential-socket\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.697899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.697613    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istiod-ca-cert\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.697899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.697667    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-token\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.697899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.697724    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-data\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.701616 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.701594    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"]
Oct 31 18:31:17.750633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.750606    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e4c133ca-openshift-default-dockercfg-d9ssj\""
Oct 31 18:31:17.795381 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.795348    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:17.798554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.798530    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-certs\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.798630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.798571    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lznrc\" (UniqueName: \"kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-kube-api-access-lznrc\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.798630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.798602    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-podinfo\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.798738 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.798651    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-envoy\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.798738 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.798679    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-socket\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.798738 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.798702    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-credential-socket\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.798887 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.798755    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istiod-ca-cert\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.798887 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.798781    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-token\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.798887 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.798806    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-data\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.799028 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.798993    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-certs\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.799166 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.799147    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-data\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.799236 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.799194    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-credential-socket\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.799676 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.799651    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istiod-ca-cert\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.799989 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.799968    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-socket\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.801055 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.801031    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-envoy\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.801306 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.801288    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-podinfo\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.856935 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.856905    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-token\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.857119 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.857098    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lznrc\" (UniqueName: \"kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-kube-api-access-lznrc\") pod \"e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") " pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:17.941466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:17.941383    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:18.051345 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:18.051304    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:18.211971 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:18.211898    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:18.370666 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:18.370636    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"]
Oct 31 18:31:18.410796 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:31:18.410772    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe4dff5_13c8_486a_8c2f_80d8cc40fe71.slice/crio-9163ec058ae8313733e76747fa4bcdff881b153f6b008565e70586573c99d704 WatchSource:0}: Error finding container 9163ec058ae8313733e76747fa4bcdff881b153f6b008565e70586573c99d704: Status 404 returned error can't find the container with id 9163ec058ae8313733e76747fa4bcdff881b153f6b008565e70586573c99d704
Oct 31 18:31:18.412775 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:18.412740    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:31:18.412863 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:18.412807    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:31:18.412863 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:18.412836    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:31:18.483504 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:18.483386    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:18.535452 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:18.535409    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" event={"ID":"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71","Type":"ContainerStarted","Data":"596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563"}
Oct 31 18:31:18.535652 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:18.535516    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" event={"ID":"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71","Type":"ContainerStarted","Data":"9163ec058ae8313733e76747fa4bcdff881b153f6b008565e70586573c99d704"}
Oct 31 18:31:18.659466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:18.659412    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" podStartSLOduration=1.65939469 podStartE2EDuration="1.65939469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:18.656184966 +0000 UTC m=+624.802634841" watchObservedRunningTime="2026-10-31 18:31:18.65939469 +0000 UTC m=+624.805844553"
Oct 31 18:31:18.941914 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:18.941889    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:19.650375 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.650345    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"]
Oct 31 18:31:19.653761 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.653747    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.717262 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.717228    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9k8k\" (UniqueName: \"kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-kube-api-access-c9k8k\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.717441 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.717282    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-socket\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.717441 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.717352    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-data\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.717441 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.717390    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-credential-socket\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.717441 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.717434    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-podinfo\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.717658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.717497    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-envoy\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.717658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.717520    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-token\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.717658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.717564    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istiod-ca-cert\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.717658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.717592    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-certs\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.735139 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.735115    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"]
Oct 31 18:31:19.776844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.776810    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:19.782570 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.782549    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8d3cca1f-openshift-default-dockercfg-g744t\""
Oct 31 18:31:19.818250 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818222    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-data\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.818430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818260    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-credential-socket\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.818430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818285    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-podinfo\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.818430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818317    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-envoy\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.818430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818340    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-token\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.818430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818379    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istiod-ca-cert\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.818430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818408    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-certs\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.818802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818446    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9k8k\" (UniqueName: \"kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-kube-api-access-c9k8k\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.818802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818492    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-socket\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.818802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818724    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-data\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.818802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818733    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-credential-socket\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.818998 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818815    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-socket\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.818998 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.818960    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-certs\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.819232 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.819212    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istiod-ca-cert\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.821159 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.821133    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-envoy\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.821336 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.821311    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-podinfo\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.876462 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.876438    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-token\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.876642 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.876599    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9k8k\" (UniqueName: \"kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-kube-api-access-c9k8k\") pod \"e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") " pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:19.947248 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.947192    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:19.966327 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:19.966303    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:20.028779 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.028732    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:20.403822 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.403798    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"]
Oct 31 18:31:20.405186 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:31:20.405154    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeec46bd_b811_4263_b2f4_ba6f4f6d8ffe.slice/crio-23f970af9016109a98b75d633ba76d63e9e3ac267d150298c53cf1a4a3ff02c3 WatchSource:0}: Error finding container 23f970af9016109a98b75d633ba76d63e9e3ac267d150298c53cf1a4a3ff02c3: Status 404 returned error can't find the container with id 23f970af9016109a98b75d633ba76d63e9e3ac267d150298c53cf1a4a3ff02c3
Oct 31 18:31:20.407649 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.407620    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:31:20.407716 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.407680    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:31:20.407716 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.407708    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:31:20.547317 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.547283    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" event={"ID":"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe","Type":"ContainerStarted","Data":"07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3"}
Oct 31 18:31:20.547516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.547343    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" event={"ID":"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe","Type":"ContainerStarted","Data":"23f970af9016109a98b75d633ba76d63e9e3ac267d150298c53cf1a4a3ff02c3"}
Oct 31 18:31:20.547516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.547390    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:20.548837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.548814    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:20.558821 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.558797    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:20.669998 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.669879    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" podStartSLOduration=1.669860592 podStartE2EDuration="1.669860592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:20.668813786 +0000 UTC m=+626.815263650" watchObservedRunningTime="2026-10-31 18:31:20.669860592 +0000 UTC m=+626.816310457"
Oct 31 18:31:20.734535 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.734500    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:20.896527 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.896431    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:20.967067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:20.966995    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:21.496785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:21.496743    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:21.664280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:21.664249    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:21.971589 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:21.971523    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:22.556941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:22.556913    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:22.557938 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:22.557917    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:31:23.405986 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:23.405946    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:23.683853 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:23.683774    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:23.844199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:23.844165    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:24.011999 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.011964    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:24.181549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.181523    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"]
Oct 31 18:31:24.185201 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.185184    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"
Oct 31 18:31:24.238120 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.238089    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bbf5caaf\""
Oct 31 18:31:24.243046 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.243025    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"]
Oct 31 18:31:24.261246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.261210    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8dlk\" (UniqueName: \"kubernetes.io/projected/1a356ee7-2002-47b0-8199-531af545cf1b-kube-api-access-c8dlk\") pod \"payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d\" (UID: \"1a356ee7-2002-47b0-8199-531af545cf1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"
Oct 31 18:31:24.261646 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.261627    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a356ee7-2002-47b0-8199-531af545cf1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d\" (UID: \"1a356ee7-2002-47b0-8199-531af545cf1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"
Oct 31 18:31:24.304685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.304600    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"]
Oct 31 18:31:24.309434 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.309409    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"
Oct 31 18:31:24.327221 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.327201    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bbf5caaf-dockercfg-c76jk\""
Oct 31 18:31:24.347928 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.347899    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:24.363906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.363205    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq7qt\" (UniqueName: \"kubernetes.io/projected/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-kube-api-access-pq7qt\") pod \"payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f\" (UID: \"a10b847c-1752-49bd-b16d-6f4f2a3a29f1\") " pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"
Oct 31 18:31:24.363906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.363272    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a356ee7-2002-47b0-8199-531af545cf1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d\" (UID: \"1a356ee7-2002-47b0-8199-531af545cf1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"
Oct 31 18:31:24.363906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.363350    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8dlk\" (UniqueName: \"kubernetes.io/projected/1a356ee7-2002-47b0-8199-531af545cf1b-kube-api-access-c8dlk\") pod \"payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d\" (UID: \"1a356ee7-2002-47b0-8199-531af545cf1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"
Oct 31 18:31:24.363906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.363394    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-plugins-config-volume\") pod \"payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f\" (UID: \"a10b847c-1752-49bd-b16d-6f4f2a3a29f1\") " pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"
Oct 31 18:31:24.366086 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.366060    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"]
Oct 31 18:31:24.367066 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.366970    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a356ee7-2002-47b0-8199-531af545cf1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d\" (UID: \"1a356ee7-2002-47b0-8199-531af545cf1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"
Oct 31 18:31:24.427427 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.427388    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8dlk\" (UniqueName: \"kubernetes.io/projected/1a356ee7-2002-47b0-8199-531af545cf1b-kube-api-access-c8dlk\") pod \"payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d\" (UID: \"1a356ee7-2002-47b0-8199-531af545cf1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"
Oct 31 18:31:24.464096 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.464067    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7qt\" (UniqueName: \"kubernetes.io/projected/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-kube-api-access-pq7qt\") pod \"payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f\" (UID: \"a10b847c-1752-49bd-b16d-6f4f2a3a29f1\") " pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"
Oct 31 18:31:24.464205 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.464121    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-plugins-config-volume\") pod \"payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f\" (UID: \"a10b847c-1752-49bd-b16d-6f4f2a3a29f1\") " pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"
Oct 31 18:31:24.465029 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.464989    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-plugins-config-volume\") pod \"payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f\" (UID: \"a10b847c-1752-49bd-b16d-6f4f2a3a29f1\") " pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"
Oct 31 18:31:24.498654 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.498631    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"
Oct 31 18:31:24.522642 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.522612    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7qt\" (UniqueName: \"kubernetes.io/projected/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-kube-api-access-pq7qt\") pod \"payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f\" (UID: \"a10b847c-1752-49bd-b16d-6f4f2a3a29f1\") " pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"
Oct 31 18:31:24.557489 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.557390    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:24.622055 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.622030    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"
Oct 31 18:31:24.870834 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.870811    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"]
Oct 31 18:31:24.871793 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:31:24.871772    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a356ee7_2002_47b0_8199_531af545cf1b.slice/crio-d2da2b0fb90bb1485882961289b6bc771e23d7181f5e7fd37f4a060539ee4183 WatchSource:0}: Error finding container d2da2b0fb90bb1485882961289b6bc771e23d7181f5e7fd37f4a060539ee4183: Status 404 returned error can't find the container with id d2da2b0fb90bb1485882961289b6bc771e23d7181f5e7fd37f4a060539ee4183
Oct 31 18:31:24.993920 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:24.993896    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"]
Oct 31 18:31:24.995498 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:31:24.995456    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10b847c_1752_49bd_b16d_6f4f2a3a29f1.slice/crio-af1d776bd5b7db1abd57c09473c6a8ecb8ed3f6a58affa17137da423734f8781 WatchSource:0}: Error finding container af1d776bd5b7db1abd57c09473c6a8ecb8ed3f6a58affa17137da423734f8781: Status 404 returned error can't find the container with id af1d776bd5b7db1abd57c09473c6a8ecb8ed3f6a58affa17137da423734f8781
Oct 31 18:31:25.214184 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.214093    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:25.390199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.390148    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:25.467433 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:31:25.467358    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:31:25.573900 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.573862    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:25.577695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.577658    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d" event={"ID":"1a356ee7-2002-47b0-8199-531af545cf1b","Type":"ContainerStarted","Data":"98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd"}
Oct 31 18:31:25.577852 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.577713    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d" event={"ID":"1a356ee7-2002-47b0-8199-531af545cf1b","Type":"ContainerStarted","Data":"d2da2b0fb90bb1485882961289b6bc771e23d7181f5e7fd37f4a060539ee4183"}
Oct 31 18:31:25.577852 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.577826    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"
Oct 31 18:31:25.579055 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.579032    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f" event={"ID":"a10b847c-1752-49bd-b16d-6f4f2a3a29f1","Type":"ContainerStarted","Data":"1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d"}
Oct 31 18:31:25.579185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.579079    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f" event={"ID":"a10b847c-1752-49bd-b16d-6f4f2a3a29f1","Type":"ContainerStarted","Data":"af1d776bd5b7db1abd57c09473c6a8ecb8ed3f6a58affa17137da423734f8781"}
Oct 31 18:31:25.579185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.579127    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"
Oct 31 18:31:25.815105 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.815028    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f" podStartSLOduration=1.8150092789999999 podStartE2EDuration="1.815009279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:25.813749876 +0000 UTC m=+631.960199741" watchObservedRunningTime="2026-10-31 18:31:25.815009279 +0000 UTC m=+631.961459144"
Oct 31 18:31:25.815871 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.815829    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d" podStartSLOduration=1.815813977 podStartE2EDuration="1.815813977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:25.694280791 +0000 UTC m=+631.840730654" watchObservedRunningTime="2026-10-31 18:31:25.815813977 +0000 UTC m=+631.962263843"
Oct 31 18:31:25.817778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.817748    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:25.972914 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:25.972878    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:26.922933 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:26.922904    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:27.129315 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:27.129281    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:27.295230 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:27.295199    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:28.567565 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:28.567527    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:28.741226 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:28.741194    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:28.944464 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:28.944396    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"]
Oct 31 18:31:28.948069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:28.948054    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"
Oct 31 18:31:29.004263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.004238    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e4c133ca\""
Oct 31 18:31:29.004400 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.004301    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e4c133ca-dockercfg-sb6vk\""
Oct 31 18:31:29.009514 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.008872    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2765h\" (UniqueName: \"kubernetes.io/projected/f618c377-4b89-4635-a1a6-07d27f5d7bf8-kube-api-access-2765h\") pod \"payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk\" (UID: \"f618c377-4b89-4635-a1a6-07d27f5d7bf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"
Oct 31 18:31:29.009514 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.009062    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f618c377-4b89-4635-a1a6-07d27f5d7bf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk\" (UID: \"f618c377-4b89-4635-a1a6-07d27f5d7bf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"
Oct 31 18:31:29.056857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.056825    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"]
Oct 31 18:31:29.108671 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.108643    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"]
Oct 31 18:31:29.109924 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.109897    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f618c377-4b89-4635-a1a6-07d27f5d7bf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk\" (UID: \"f618c377-4b89-4635-a1a6-07d27f5d7bf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"
Oct 31 18:31:29.110083 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.109963    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2765h\" (UniqueName: \"kubernetes.io/projected/f618c377-4b89-4635-a1a6-07d27f5d7bf8-kube-api-access-2765h\") pod \"payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk\" (UID: \"f618c377-4b89-4635-a1a6-07d27f5d7bf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"
Oct 31 18:31:29.110604 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.110586    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f618c377-4b89-4635-a1a6-07d27f5d7bf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk\" (UID: \"f618c377-4b89-4635-a1a6-07d27f5d7bf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"
Oct 31 18:31:29.112319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.112301    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"
Oct 31 18:31:29.170055 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.170017    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"]
Oct 31 18:31:29.178903 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.171781    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2765h\" (UniqueName: \"kubernetes.io/projected/f618c377-4b89-4635-a1a6-07d27f5d7bf8-kube-api-access-2765h\") pod \"payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk\" (UID: \"f618c377-4b89-4635-a1a6-07d27f5d7bf8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"
Oct 31 18:31:29.211301 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.211234    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c69198a-3c8f-4254-955c-6738b7769bab-plugins-config-volume\") pod \"payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj\" (UID: \"1c69198a-3c8f-4254-955c-6738b7769bab\") " pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"
Oct 31 18:31:29.211301 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.211277    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2z8q\" (UniqueName: \"kubernetes.io/projected/1c69198a-3c8f-4254-955c-6738b7769bab-kube-api-access-m2z8q\") pod \"payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj\" (UID: \"1c69198a-3c8f-4254-955c-6738b7769bab\") " pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"
Oct 31 18:31:29.261550 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.261517    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"
Oct 31 18:31:29.312139 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.312113    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c69198a-3c8f-4254-955c-6738b7769bab-plugins-config-volume\") pod \"payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj\" (UID: \"1c69198a-3c8f-4254-955c-6738b7769bab\") " pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"
Oct 31 18:31:29.312357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.312162    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2z8q\" (UniqueName: \"kubernetes.io/projected/1c69198a-3c8f-4254-955c-6738b7769bab-kube-api-access-m2z8q\") pod \"payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj\" (UID: \"1c69198a-3c8f-4254-955c-6738b7769bab\") " pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"
Oct 31 18:31:29.312915 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.312889    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c69198a-3c8f-4254-955c-6738b7769bab-plugins-config-volume\") pod \"payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj\" (UID: \"1c69198a-3c8f-4254-955c-6738b7769bab\") " pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"
Oct 31 18:31:29.370053 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.370019    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2z8q\" (UniqueName: \"kubernetes.io/projected/1c69198a-3c8f-4254-955c-6738b7769bab-kube-api-access-m2z8q\") pod \"payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj\" (UID: \"1c69198a-3c8f-4254-955c-6738b7769bab\") " pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"
Oct 31 18:31:29.425573 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.425544    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"
Oct 31 18:31:29.634802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.634781    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"]
Oct 31 18:31:29.636103 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:31:29.636080    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf618c377_4b89_4635_a1a6_07d27f5d7bf8.slice/crio-ee37d544af97459c9ad5b271116d4db77502af7f9b2383c082ea724c30209de8 WatchSource:0}: Error finding container ee37d544af97459c9ad5b271116d4db77502af7f9b2383c082ea724c30209de8: Status 404 returned error can't find the container with id ee37d544af97459c9ad5b271116d4db77502af7f9b2383c082ea724c30209de8
Oct 31 18:31:29.794638 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:29.794614    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"]
Oct 31 18:31:29.796309 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:31:29.796282    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c69198a_3c8f_4254_955c_6738b7769bab.slice/crio-3e361be48bf5b1b1e2f4f23b2b5bc7432c8415a21588cd03d913e89c09e0ae46 WatchSource:0}: Error finding container 3e361be48bf5b1b1e2f4f23b2b5bc7432c8415a21588cd03d913e89c09e0ae46: Status 404 returned error can't find the container with id 3e361be48bf5b1b1e2f4f23b2b5bc7432c8415a21588cd03d913e89c09e0ae46
Oct 31 18:31:30.603616 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:30.603575    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk" event={"ID":"f618c377-4b89-4635-a1a6-07d27f5d7bf8","Type":"ContainerStarted","Data":"0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed"}
Oct 31 18:31:30.603860 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:30.603642    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk" event={"ID":"f618c377-4b89-4635-a1a6-07d27f5d7bf8","Type":"ContainerStarted","Data":"ee37d544af97459c9ad5b271116d4db77502af7f9b2383c082ea724c30209de8"}
Oct 31 18:31:30.603860 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:30.603713    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"
Oct 31 18:31:30.605364 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:30.605341    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj" event={"ID":"1c69198a-3c8f-4254-955c-6738b7769bab","Type":"ContainerStarted","Data":"225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a"}
Oct 31 18:31:30.605483 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:30.605381    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj" event={"ID":"1c69198a-3c8f-4254-955c-6738b7769bab","Type":"ContainerStarted","Data":"3e361be48bf5b1b1e2f4f23b2b5bc7432c8415a21588cd03d913e89c09e0ae46"}
Oct 31 18:31:30.605483 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:30.605400    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"
Oct 31 18:31:30.735029 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:30.735002    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:30.888931 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:30.888843    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk" podStartSLOduration=2.88882821 podStartE2EDuration="2.88882821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:30.769293799 +0000 UTC m=+636.915743663" watchObservedRunningTime="2026-10-31 18:31:30.88882821 +0000 UTC m=+637.035278072"
Oct 31 18:31:30.888931 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:30.888921    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj" podStartSLOduration=1.888917811 podStartE2EDuration="1.888917811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:30.887801223 +0000 UTC m=+637.034251088" watchObservedRunningTime="2026-10-31 18:31:30.888917811 +0000 UTC m=+637.035367674"
Oct 31 18:31:30.935256 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:30.935222    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:31.110762 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.110713    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:31.288110 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.288068    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:31.387169 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.387141    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"]
Oct 31 18:31:31.390885 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.390870    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"
Oct 31 18:31:31.446379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.446353    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8d3cca1f\""
Oct 31 18:31:31.446529 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.446357    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8d3cca1f-dockercfg-vvs4v\""
Oct 31 18:31:31.499033 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.499001    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"]
Oct 31 18:31:31.535241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.535201    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a09f2c6-e58f-436b-a794-63b0da9037fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg\" (UID: \"9a09f2c6-e58f-436b-a794-63b0da9037fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"
Oct 31 18:31:31.535420 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.535284    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7fv\" (UniqueName: \"kubernetes.io/projected/9a09f2c6-e58f-436b-a794-63b0da9037fb-kube-api-access-wd7fv\") pod \"payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg\" (UID: \"9a09f2c6-e58f-436b-a794-63b0da9037fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"
Oct 31 18:31:31.549916 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.549851    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"]
Oct 31 18:31:31.553403 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.553388    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"
Oct 31 18:31:31.610608 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.610578    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"]
Oct 31 18:31:31.636774 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.636735    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13097b93-539f-4187-86b4-10b649a9ef85-plugins-config-volume\") pod \"payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg\" (UID: \"13097b93-539f-4187-86b4-10b649a9ef85\") " pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"
Oct 31 18:31:31.636923 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.636792    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a09f2c6-e58f-436b-a794-63b0da9037fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg\" (UID: \"9a09f2c6-e58f-436b-a794-63b0da9037fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"
Oct 31 18:31:31.636923 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.636865    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bpp4\" (UniqueName: \"kubernetes.io/projected/13097b93-539f-4187-86b4-10b649a9ef85-kube-api-access-2bpp4\") pod \"payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg\" (UID: \"13097b93-539f-4187-86b4-10b649a9ef85\") " pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"
Oct 31 18:31:31.636999 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.636985    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7fv\" (UniqueName: \"kubernetes.io/projected/9a09f2c6-e58f-436b-a794-63b0da9037fb-kube-api-access-wd7fv\") pod \"payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg\" (UID: \"9a09f2c6-e58f-436b-a794-63b0da9037fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"
Oct 31 18:31:31.637530 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.637509    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a09f2c6-e58f-436b-a794-63b0da9037fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg\" (UID: \"9a09f2c6-e58f-436b-a794-63b0da9037fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"
Oct 31 18:31:31.693995 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.693972    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7fv\" (UniqueName: \"kubernetes.io/projected/9a09f2c6-e58f-436b-a794-63b0da9037fb-kube-api-access-wd7fv\") pod \"payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg\" (UID: \"9a09f2c6-e58f-436b-a794-63b0da9037fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"
Oct 31 18:31:31.703844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.703819    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"
Oct 31 18:31:31.738206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.738170    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bpp4\" (UniqueName: \"kubernetes.io/projected/13097b93-539f-4187-86b4-10b649a9ef85-kube-api-access-2bpp4\") pod \"payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg\" (UID: \"13097b93-539f-4187-86b4-10b649a9ef85\") " pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"
Oct 31 18:31:31.738661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.738319    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13097b93-539f-4187-86b4-10b649a9ef85-plugins-config-volume\") pod \"payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg\" (UID: \"13097b93-539f-4187-86b4-10b649a9ef85\") " pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"
Oct 31 18:31:31.739066 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.739039    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13097b93-539f-4187-86b4-10b649a9ef85-plugins-config-volume\") pod \"payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg\" (UID: \"13097b93-539f-4187-86b4-10b649a9ef85\") " pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"
Oct 31 18:31:31.795451 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.795412    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bpp4\" (UniqueName: \"kubernetes.io/projected/13097b93-539f-4187-86b4-10b649a9ef85-kube-api-access-2bpp4\") pod \"payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg\" (UID: \"13097b93-539f-4187-86b4-10b649a9ef85\") " pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"
Oct 31 18:31:31.866305 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:31.866223    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"
Oct 31 18:31:32.075732 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:32.075709    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"]
Oct 31 18:31:32.078673 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:31:32.078634    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a09f2c6_e58f_436b_a794_63b0da9037fb.slice/crio-4d647be6f3da187a0dd2e550db0d9228541c816499027b1f8b6d829f6ac63b23 WatchSource:0}: Error finding container 4d647be6f3da187a0dd2e550db0d9228541c816499027b1f8b6d829f6ac63b23: Status 404 returned error can't find the container with id 4d647be6f3da187a0dd2e550db0d9228541c816499027b1f8b6d829f6ac63b23
Oct 31 18:31:32.144955 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:32.144931    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:32.238553 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:32.238524    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"]
Oct 31 18:31:32.240255 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:31:32.240230    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13097b93_539f_4187_86b4_10b649a9ef85.slice/crio-73472ccf66428849a047c9163d63f6ab3fc139890d415cf0bc9aec8b1117b799 WatchSource:0}: Error finding container 73472ccf66428849a047c9163d63f6ab3fc139890d415cf0bc9aec8b1117b799: Status 404 returned error can't find the container with id 73472ccf66428849a047c9163d63f6ab3fc139890d415cf0bc9aec8b1117b799
Oct 31 18:31:32.350238 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:32.350203    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:32.618392 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:32.618346    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg" event={"ID":"9a09f2c6-e58f-436b-a794-63b0da9037fb","Type":"ContainerStarted","Data":"a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86"}
Oct 31 18:31:32.618583 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:32.618452    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg" event={"ID":"9a09f2c6-e58f-436b-a794-63b0da9037fb","Type":"ContainerStarted","Data":"4d647be6f3da187a0dd2e550db0d9228541c816499027b1f8b6d829f6ac63b23"}
Oct 31 18:31:32.618583 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:32.618498    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"
Oct 31 18:31:32.619945 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:32.619922    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg" event={"ID":"13097b93-539f-4187-86b4-10b649a9ef85","Type":"ContainerStarted","Data":"e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a"}
Oct 31 18:31:32.620037 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:32.619961    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg" event={"ID":"13097b93-539f-4187-86b4-10b649a9ef85","Type":"ContainerStarted","Data":"73472ccf66428849a047c9163d63f6ab3fc139890d415cf0bc9aec8b1117b799"}
Oct 31 18:31:32.620076 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:32.620044    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"
Oct 31 18:31:32.735161 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:32.735114    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg" podStartSLOduration=1.73510032 podStartE2EDuration="1.73510032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:32.733408575 +0000 UTC m=+638.879858440" watchObservedRunningTime="2026-10-31 18:31:32.73510032 +0000 UTC m=+638.881550183"
Oct 31 18:31:32.850424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:32.850355    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg" podStartSLOduration=1.850336413 podStartE2EDuration="1.850336413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:32.847536507 +0000 UTC m=+638.993986370" watchObservedRunningTime="2026-10-31 18:31:32.850336413 +0000 UTC m=+638.996786277"
Oct 31 18:31:33.529592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:33.529558    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:33.738196 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:33.738162    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:35.613115 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:35.613082    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:31:36.587370 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:36.587339    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"
Oct 31 18:31:36.588220 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:36.588197    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"
Oct 31 18:31:37.320834 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:37.320772    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:37.524994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:37.524957    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:37.685552 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:37.682858    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:41.614518 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:41.614492    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"
Oct 31 18:31:41.614935 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:41.614583    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"
Oct 31 18:31:41.867253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:41.867189    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:42.969152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:42.969114    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:43.181742 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:43.181700    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:43.630146 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:43.630119    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"
Oct 31 18:31:43.630345 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:43.630180    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"
Oct 31 18:31:43.694994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:43.694965    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:44.488455 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:44.488426    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:31:45.697689 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:45.697657    4484 generic.go:356] "Generic (PLEG): container finished" podID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerID="596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563" exitCode=0
Oct 31 18:31:45.698209 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:45.697738    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" event={"ID":"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71","Type":"ContainerDied","Data":"596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563"}
Oct 31 18:31:45.698209 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:45.698111    4484 scope.go:122] "RemoveContainer" containerID="596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563"
Oct 31 18:31:45.699408 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:45.699382    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:31:45.699496 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:45.699447    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:31:45.699548 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:45.699501    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:31:46.532205 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:46.532166    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:46.686199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:46.686151    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:46.704840 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:46.704805    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" event={"ID":"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71","Type":"ContainerStarted","Data":"ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343"}
Oct 31 18:31:46.942020 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:46.941932    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:46.943382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:46.943355    4484 patch_prober.go:29] interesting pod/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 18:31:46.943517 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:46.943422    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 18:31:47.942171 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:47.942128    4484 patch_prober.go:29] interesting pod/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 18:31:47.942639 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:47.942191    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 18:31:48.942595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:48.942559    4484 patch_prober.go:29] interesting pod/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 18:31:48.942977 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:48.942607    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 18:31:49.942146 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:49.942114    4484 patch_prober.go:29] interesting pod/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 18:31:49.942345 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:49.942173    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 18:31:50.548273 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:50.548236    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:50.942380 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:50.942298    4484 patch_prober.go:29] interesting pod/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 18:31:50.942380 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:50.942358    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 18:31:51.350886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:51.350851    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:51.946390 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:51.946353    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:51.947424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:51.947402    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:31:55.470893 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:31:55.470853    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:31:55.700641 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.700610    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"]
Oct 31 18:31:55.704450 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.704434    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:55.808834 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.808809    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8d3cca1f-serving-cert\""
Oct 31 18:31:55.809088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.809068    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 18:31:55.809168 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.809066    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czkfc\""
Oct 31 18:31:55.811030 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.811008    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"]
Oct 31 18:31:55.888359 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.888324    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-metrics-certs\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:55.888558 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.888393    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzxs\" (UniqueName: \"kubernetes.io/projected/66c1aebe-1457-45d8-bf58-0193185f1f3e-kube-api-access-hvzxs\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:55.888558 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.888522    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c1aebe-1457-45d8-bf58-0193185f1f3e-openshift-service-ca\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:55.888678 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.888648    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-maas-api-tls\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:55.915344 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.915308    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:55.989154 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.989122    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-maas-api-tls\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:55.989518 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.989199    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-metrics-certs\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:55.989518 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:31:55.989298    4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-serving-cert: secret "maas-api-e2e-ait-8d3cca1f-serving-cert" not found
Oct 31 18:31:55.989518 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:31:55.989374    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-maas-api-tls podName:66c1aebe-1457-45d8-bf58-0193185f1f3e nodeName:}" failed. No retries permitted until 2026-10-31 18:31:56.489351364 +0000 UTC m=+662.635801231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-maas-api-tls") pod "maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt" (UID: "66c1aebe-1457-45d8-bf58-0193185f1f3e") : secret "maas-api-e2e-ait-8d3cca1f-serving-cert" not found
Oct 31 18:31:55.989858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.989831    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzxs\" (UniqueName: \"kubernetes.io/projected/66c1aebe-1457-45d8-bf58-0193185f1f3e-kube-api-access-hvzxs\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:55.989939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.989914    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c1aebe-1457-45d8-bf58-0193185f1f3e-openshift-service-ca\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:55.991045 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.991021    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c1aebe-1457-45d8-bf58-0193185f1f3e-openshift-service-ca\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:55.993144 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:55.993121    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-metrics-certs\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:56.047213 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:56.047180    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzxs\" (UniqueName: \"kubernetes.io/projected/66c1aebe-1457-45d8-bf58-0193185f1f3e-kube-api-access-hvzxs\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:56.073311 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:56.073229    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:56.221907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:56.221873    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:31:56.496691 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:56.496664    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-maas-api-tls\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:56.499248 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:56.499217    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-maas-api-tls\") pod \"maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:56.617734 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:56.617701    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:56.990349 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:56.990327    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"]
Oct 31 18:31:56.991783 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:31:56.991759    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c1aebe_1457_45d8_bf58_0193185f1f3e.slice/crio-d0808b66bcc9bfa05581ba4aa4776ade629eaaa0291a5b6a452c5c5c52e97bf3 WatchSource:0}: Error finding container d0808b66bcc9bfa05581ba4aa4776ade629eaaa0291a5b6a452c5c5c52e97bf3: Status 404 returned error can't find the container with id d0808b66bcc9bfa05581ba4aa4776ade629eaaa0291a5b6a452c5c5c52e97bf3
Oct 31 18:31:56.993249 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:56.993229    4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:31:57.763306 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:57.763269    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt" event={"ID":"66c1aebe-1457-45d8-bf58-0193185f1f3e","Type":"ContainerStarted","Data":"d0808b66bcc9bfa05581ba4aa4776ade629eaaa0291a5b6a452c5c5c52e97bf3"}
Oct 31 18:31:59.776293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:59.776253    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt" event={"ID":"66c1aebe-1457-45d8-bf58-0193185f1f3e","Type":"ContainerStarted","Data":"22d458a9f0792914478ffd8442f8df350712784d942f97cd72fa23c350aca5a9"}
Oct 31 18:31:59.776787 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:59.776344    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:31:59.894265 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:31:59.894210    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt" podStartSLOduration=2.984164104 podStartE2EDuration="4.894194963s" totalImagesPullingTime="1.910030859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:31:59.89369902 +0000 UTC m=+666.040148885" watchObservedRunningTime="2026-10-31 18:31:59.894194963 +0000 UTC m=+666.040644827"
Oct 31 18:32:01.292077 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:01.292043    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:03.525760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:03.525721    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:05.788107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:05.788030    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:32:06.075365 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:06.075280    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:32:06.395906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:06.395824    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:11.193768 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:11.193733    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:12.931824 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:12.931784    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"]
Oct 31 18:32:13.037821 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.037783    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"]
Oct 31 18:32:13.175947 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.175907    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"]
Oct 31 18:32:13.179860 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.179833    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:13.235352 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.235124    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:32:13.235352 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.235319    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:32:13.238136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.238101    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"]
Oct 31 18:32:13.263266 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.263233    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92\" (UID: \"f0570551-f242-44dd-a559-dd3d4daed62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:13.263266 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.263276    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92\" (UID: \"f0570551-f242-44dd-a559-dd3d4daed62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:13.263517 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.263366    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrvx\" (UniqueName: \"kubernetes.io/projected/f0570551-f242-44dd-a559-dd3d4daed62d-kube-api-access-fgrvx\") pod \"maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92\" (UID: \"f0570551-f242-44dd-a559-dd3d4daed62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:13.319318 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.319276    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:13.364933 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.364889    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrvx\" (UniqueName: \"kubernetes.io/projected/f0570551-f242-44dd-a559-dd3d4daed62d-kube-api-access-fgrvx\") pod \"maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92\" (UID: \"f0570551-f242-44dd-a559-dd3d4daed62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:13.365137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.365073    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92\" (UID: \"f0570551-f242-44dd-a559-dd3d4daed62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:13.365137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.365114    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92\" (UID: \"f0570551-f242-44dd-a559-dd3d4daed62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:13.366068 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.366030    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92\" (UID: \"f0570551-f242-44dd-a559-dd3d4daed62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:13.367887 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.367859    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92\" (UID: \"f0570551-f242-44dd-a559-dd3d4daed62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:13.423949 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.423909    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrvx\" (UniqueName: \"kubernetes.io/projected/f0570551-f242-44dd-a559-dd3d4daed62d-kube-api-access-fgrvx\") pod \"maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92\" (UID: \"f0570551-f242-44dd-a559-dd3d4daed62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:13.469791 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.469703    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:13.494583 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.494542    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:13.875346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:13.875316    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"]
Oct 31 18:32:13.876267 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:32:13.876239    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0570551_f242_44dd_a559_dd3d4daed62d.slice/crio-8d3801aa6f8d9faefcc28c1ecd6f7445141b0d65780e0cfb013cefa59cba7d74 WatchSource:0}: Error finding container 8d3801aa6f8d9faefcc28c1ecd6f7445141b0d65780e0cfb013cefa59cba7d74: Status 404 returned error can't find the container with id 8d3801aa6f8d9faefcc28c1ecd6f7445141b0d65780e0cfb013cefa59cba7d74
Oct 31 18:32:14.859898 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:14.859868    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92" event={"ID":"f0570551-f242-44dd-a559-dd3d4daed62d","Type":"ContainerStarted","Data":"8d3801aa6f8d9faefcc28c1ecd6f7445141b0d65780e0cfb013cefa59cba7d74"}
Oct 31 18:32:16.470874 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:16.470831    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:16.871660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:16.871626    4484 generic.go:356] "Generic (PLEG): container finished" podID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerID="07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3" exitCode=0
Oct 31 18:32:16.871865 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:16.871696    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" event={"ID":"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe","Type":"ContainerDied","Data":"07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3"}
Oct 31 18:32:16.872191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:16.872173    4484 scope.go:122] "RemoveContainer" containerID="07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3"
Oct 31 18:32:16.873985 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:16.873955    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:32:16.874206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:16.874039    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:32:16.874206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:16.874077    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:32:17.883543 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:17.882948    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" event={"ID":"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe","Type":"ContainerStarted","Data":"a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013"}
Oct 31 18:32:17.932305 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:17.932256    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj" podUID="1c69198a-3c8f-4254-955c-6738b7769bab" containerName="payload-processing" containerID="cri-o://225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a" gracePeriod=25
Oct 31 18:32:17.982627 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:17.973921    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:32:17.983405 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:17.982979    4484 patch_prober.go:29] interesting pod/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 31 18:32:17.983405 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:17.983078    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 31 18:32:18.038654 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.038610    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk" podUID="f618c377-4b89-4635-a1a6-07d27f5d7bf8" containerName="payload-pre-processing" containerID="cri-o://0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed" gracePeriod=25
Oct 31 18:32:18.309422 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.302649    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"
Oct 31 18:32:18.413029 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.412990    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"
Oct 31 18:32:18.427294 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.427254    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c69198a-3c8f-4254-955c-6738b7769bab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c69198a-3c8f-4254-955c-6738b7769bab-plugins-config-volume\") pod \"1c69198a-3c8f-4254-955c-6738b7769bab\" (UID: \"1c69198a-3c8f-4254-955c-6738b7769bab\") "
Oct 31 18:32:18.427495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.427311    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c69198a-3c8f-4254-955c-6738b7769bab-kube-api-access-m2z8q\" (UniqueName: \"kubernetes.io/projected/1c69198a-3c8f-4254-955c-6738b7769bab-kube-api-access-m2z8q\") pod \"1c69198a-3c8f-4254-955c-6738b7769bab\" (UID: \"1c69198a-3c8f-4254-955c-6738b7769bab\") "
Oct 31 18:32:18.428523 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.428415    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c69198a-3c8f-4254-955c-6738b7769bab-plugins-config-volume" pod "1c69198a-3c8f-4254-955c-6738b7769bab" (UID: "1c69198a-3c8f-4254-955c-6738b7769bab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:18.430404 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.430372    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c69198a-3c8f-4254-955c-6738b7769bab-kube-api-access-m2z8q" pod "1c69198a-3c8f-4254-955c-6738b7769bab" (UID: "1c69198a-3c8f-4254-955c-6738b7769bab"). InnerVolumeSpecName "kube-api-access-m2z8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:18.528947 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.528912    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f618c377-4b89-4635-a1a6-07d27f5d7bf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f618c377-4b89-4635-a1a6-07d27f5d7bf8-plugins-config-volume\") pod \"f618c377-4b89-4635-a1a6-07d27f5d7bf8\" (UID: \"f618c377-4b89-4635-a1a6-07d27f5d7bf8\") "
Oct 31 18:32:18.529129 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.529014    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f618c377-4b89-4635-a1a6-07d27f5d7bf8-kube-api-access-2765h\" (UniqueName: \"kubernetes.io/projected/f618c377-4b89-4635-a1a6-07d27f5d7bf8-kube-api-access-2765h\") pod \"f618c377-4b89-4635-a1a6-07d27f5d7bf8\" (UID: \"f618c377-4b89-4635-a1a6-07d27f5d7bf8\") "
Oct 31 18:32:18.529309 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.529281    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f618c377-4b89-4635-a1a6-07d27f5d7bf8-plugins-config-volume" pod "f618c377-4b89-4635-a1a6-07d27f5d7bf8" (UID: "f618c377-4b89-4635-a1a6-07d27f5d7bf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:18.529388 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.529375    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c69198a-3c8f-4254-955c-6738b7769bab-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:18.529428 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.529390    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2z8q\" (UniqueName: \"kubernetes.io/projected/1c69198a-3c8f-4254-955c-6738b7769bab-kube-api-access-m2z8q\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:18.529428 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.529399    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f618c377-4b89-4635-a1a6-07d27f5d7bf8-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:18.531402 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.531369    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f618c377-4b89-4635-a1a6-07d27f5d7bf8-kube-api-access-2765h" pod "f618c377-4b89-4635-a1a6-07d27f5d7bf8" (UID: "f618c377-4b89-4635-a1a6-07d27f5d7bf8"). InnerVolumeSpecName "kube-api-access-2765h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:18.630899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.630876    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2765h\" (UniqueName: \"kubernetes.io/projected/f618c377-4b89-4635-a1a6-07d27f5d7bf8-kube-api-access-2765h\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:18.892414 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.892381    4484 generic.go:356] "Generic (PLEG): container finished" podID="f618c377-4b89-4635-a1a6-07d27f5d7bf8" containerID="0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed" exitCode=0
Oct 31 18:32:18.892906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.892425    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"
Oct 31 18:32:18.892906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.892486    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk" event={"ID":"f618c377-4b89-4635-a1a6-07d27f5d7bf8","Type":"ContainerDied","Data":"0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed"}
Oct 31 18:32:18.892906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.892544    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk" event={"ID":"f618c377-4b89-4635-a1a6-07d27f5d7bf8","Type":"ContainerDied","Data":"ee37d544af97459c9ad5b271116d4db77502af7f9b2383c082ea724c30209de8"}
Oct 31 18:32:18.892906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.892569    4484 scope.go:122] "RemoveContainer" containerID="0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed"
Oct 31 18:32:18.893923 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.893901    4484 generic.go:356] "Generic (PLEG): container finished" podID="1c69198a-3c8f-4254-955c-6738b7769bab" containerID="225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a" exitCode=0
Oct 31 18:32:18.894025 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.893960    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"
Oct 31 18:32:18.894025 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.893981    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj" event={"ID":"1c69198a-3c8f-4254-955c-6738b7769bab","Type":"ContainerDied","Data":"225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a"}
Oct 31 18:32:18.894025 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.894019    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj" event={"ID":"1c69198a-3c8f-4254-955c-6738b7769bab","Type":"ContainerDied","Data":"3e361be48bf5b1b1e2f4f23b2b5bc7432c8415a21588cd03d913e89c09e0ae46"}
Oct 31 18:32:18.902006 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.901985    4484 scope.go:122] "RemoveContainer" containerID="0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed"
Oct 31 18:32:18.902288 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:32:18.902268    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed\": container with ID starting with 0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed not found: ID does not exist" containerID="0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed"
Oct 31 18:32:18.902378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.902299    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed"} err="failed to get container status \"0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed\": rpc error: code = NotFound desc = could not find container \"0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed\": container with ID starting with 0590aea48fce7a45f07c5596a50178a9521ebeb1a73c8c3cf1647f27158108ed not found: ID does not exist"
Oct 31 18:32:18.902378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.902323    4484 scope.go:122] "RemoveContainer" containerID="225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a"
Oct 31 18:32:18.911075 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.911053    4484 scope.go:122] "RemoveContainer" containerID="225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a"
Oct 31 18:32:18.911362 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:32:18.911344    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a\": container with ID starting with 225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a not found: ID does not exist" containerID="225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a"
Oct 31 18:32:18.911420 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.911369    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a"} err="failed to get container status \"225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a\": rpc error: code = NotFound desc = could not find container \"225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a\": container with ID starting with 225c4647a6a0e4b405b8004734fdae2f4ea4cc46745120fb2132a80e939eff2a not found: ID does not exist"
Oct 31 18:32:18.975498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.974986    4484 patch_prober.go:29] interesting pod/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 31 18:32:18.975498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:18.975055    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 31 18:32:19.058141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:19.058104    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"]
Oct 31 18:32:19.059901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:19.059874    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4c133ca-7fcd9fb65-8cqrj"]
Oct 31 18:32:19.187218 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:19.186942    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"]
Oct 31 18:32:19.190264 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:19.190228    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4c133ca-7449f468c-t7fgk"]
Oct 31 18:32:19.900293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:19.900200    4484 generic.go:356] "Generic (PLEG): container finished" podID="f0570551-f242-44dd-a559-dd3d4daed62d" containerID="7010cbf616913179405c33892241511d9246d6143f35c2afdcd77d25634a4c04" exitCode=0
Oct 31 18:32:19.900293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:19.900277    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92" event={"ID":"f0570551-f242-44dd-a559-dd3d4daed62d","Type":"ContainerDied","Data":"7010cbf616913179405c33892241511d9246d6143f35c2afdcd77d25634a4c04"}
Oct 31 18:32:19.974965 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:19.974928    4484 patch_prober.go:29] interesting pod/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 31 18:32:19.975246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:19.975215    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 31 18:32:19.994049 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:19.994010    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c69198a-3c8f-4254-955c-6738b7769bab" path="/var/lib/kubelet/pods/1c69198a-3c8f-4254-955c-6738b7769bab/volumes"
Oct 31 18:32:19.994440 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:19.994424    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f618c377-4b89-4635-a1a6-07d27f5d7bf8" path="/var/lib/kubelet/pods/f618c377-4b89-4635-a1a6-07d27f5d7bf8/volumes"
Oct 31 18:32:20.845756 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:20.845718    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:20.986082 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:20.985576    4484 patch_prober.go:29] interesting pod/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 31 18:32:20.986082 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:20.985642    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 31 18:32:21.094159 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.094125    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:21.159080 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.158991    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-xks-ca\") pod \"f0570551-f242-44dd-a559-dd3d4daed62d\" (UID: \"f0570551-f242-44dd-a559-dd3d4daed62d\") "
Oct 31 18:32:21.159080 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.159048    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-openshift-ca\") pod \"f0570551-f242-44dd-a559-dd3d4daed62d\" (UID: \"f0570551-f242-44dd-a559-dd3d4daed62d\") "
Oct 31 18:32:21.159273 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.159085    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0570551-f242-44dd-a559-dd3d4daed62d-kube-api-access-fgrvx\" (UniqueName: \"kubernetes.io/projected/f0570551-f242-44dd-a559-dd3d4daed62d-kube-api-access-fgrvx\") pod \"f0570551-f242-44dd-a559-dd3d4daed62d\" (UID: \"f0570551-f242-44dd-a559-dd3d4daed62d\") "
Oct 31 18:32:21.159490 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.159444    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-openshift-ca" pod "f0570551-f242-44dd-a559-dd3d4daed62d" (UID: "f0570551-f242-44dd-a559-dd3d4daed62d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:21.186179 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.186081    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0570551-f242-44dd-a559-dd3d4daed62d-kube-api-access-fgrvx" pod "f0570551-f242-44dd-a559-dd3d4daed62d" (UID: "f0570551-f242-44dd-a559-dd3d4daed62d"). InnerVolumeSpecName "kube-api-access-fgrvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:21.186622 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.186593    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-xks-ca" pod "f0570551-f242-44dd-a559-dd3d4daed62d" (UID: "f0570551-f242-44dd-a559-dd3d4daed62d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:32:21.260078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.260044    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:21.260078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.260074    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0570551-f242-44dd-a559-dd3d4daed62d-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:21.260078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.260085    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgrvx\" (UniqueName: \"kubernetes.io/projected/f0570551-f242-44dd-a559-dd3d4daed62d-kube-api-access-fgrvx\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:21.914108 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.914074    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"
Oct 31 18:32:21.914108 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.914104    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92" event={"ID":"f0570551-f242-44dd-a559-dd3d4daed62d","Type":"ContainerDied","Data":"8d3801aa6f8d9faefcc28c1ecd6f7445141b0d65780e0cfb013cefa59cba7d74"}
Oct 31 18:32:21.914345 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.914146    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d3801aa6f8d9faefcc28c1ecd6f7445141b0d65780e0cfb013cefa59cba7d74"
Oct 31 18:32:21.967447 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.966984    4484 patch_prober.go:29] interesting pod/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 31 18:32:21.967447 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:21.967048    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 31 18:32:22.430722 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:22.430685    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:22.556691 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:22.556661    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:32:22.803848 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:22.803814    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:22.960131 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:22.960094    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:22.967147 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:22.967105    4484 patch_prober.go:29] interesting pod/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 31 18:32:22.967277 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:22.967173    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 31 18:32:23.079125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.079034    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"]
Oct 31 18:32:23.079297 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.079241    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerName="istio-proxy" containerID="cri-o://ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343" gracePeriod=30
Oct 31 18:32:23.125361 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.125292    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:23.603754 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.603723    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"]
Oct 31 18:32:23.604185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.604171    4484 cpu_manager.go:414] "containerMap: removing container" podUID="1c69198a-3c8f-4254-955c-6738b7769bab" containerName="payload-processing"
Oct 31 18:32:23.604185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.604185    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c69198a-3c8f-4254-955c-6738b7769bab" containerName="payload-processing"
Oct 31 18:32:23.604274 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.604211    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f0570551-f242-44dd-a559-dd3d4daed62d" containerName="revoke-keys"
Oct 31 18:32:23.604274 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.604217    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0570551-f242-44dd-a559-dd3d4daed62d" containerName="revoke-keys"
Oct 31 18:32:23.604274 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.604227    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f618c377-4b89-4635-a1a6-07d27f5d7bf8" containerName="payload-pre-processing"
Oct 31 18:32:23.604274 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.604232    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f618c377-4b89-4635-a1a6-07d27f5d7bf8" containerName="payload-pre-processing"
Oct 31 18:32:23.604397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.604304    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0570551-f242-44dd-a559-dd3d4daed62d" containerName="revoke-keys"
Oct 31 18:32:23.604397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.604312    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f618c377-4b89-4635-a1a6-07d27f5d7bf8" containerName="payload-pre-processing"
Oct 31 18:32:23.604397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.604322    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c69198a-3c8f-4254-955c-6738b7769bab" containerName="payload-processing"
Oct 31 18:32:23.608395 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.608378    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:23.661439 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.661408    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:32:23.661439 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.661406    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:32:23.662008 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.661977    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"]
Oct 31 18:32:23.703931 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.703895    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c7a8371-kskql\" (UID: \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:23.704111 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.704051    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c7a8371-kskql\" (UID: \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:23.704153 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.704123    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp9sm\" (UniqueName: \"kubernetes.io/projected/2c6c9a2d-169b-46fc-9dea-bc46b6158003-kube-api-access-pp9sm\") pod \"maas-api-revoke-keys-e2e-derive-3c7a8371-kskql\" (UID: \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:23.805680 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.805644    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c7a8371-kskql\" (UID: \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:23.805881 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.805791    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c7a8371-kskql\" (UID: \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:23.805881 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.805841    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp9sm\" (UniqueName: \"kubernetes.io/projected/2c6c9a2d-169b-46fc-9dea-bc46b6158003-kube-api-access-pp9sm\") pod \"maas-api-revoke-keys-e2e-derive-3c7a8371-kskql\" (UID: \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:23.806505 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.806449    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c7a8371-kskql\" (UID: \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:23.808242 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.808221    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c7a8371-kskql\" (UID: \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:23.863525 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.863433    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp9sm\" (UniqueName: \"kubernetes.io/projected/2c6c9a2d-169b-46fc-9dea-bc46b6158003-kube-api-access-pp9sm\") pod \"maas-api-revoke-keys-e2e-derive-3c7a8371-kskql\" (UID: \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:23.903504 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.903435    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:23.921387 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.921356    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:23.971397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.971368    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:32:23.972784 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:23.972755    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:32:24.294986 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.294961    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"]
Oct 31 18:32:24.296458 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:32:24.296428    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6c9a2d_169b_46fc_9dea_bc46b6158003.slice/crio-868f1b2d932262e759e1b244b3ac392c11db5f56b2973d22c7ab2772c5b6df43 WatchSource:0}: Error finding container 868f1b2d932262e759e1b244b3ac392c11db5f56b2973d22c7ab2772c5b6df43: Status 404 returned error can't find the container with id 868f1b2d932262e759e1b244b3ac392c11db5f56b2973d22c7ab2772c5b6df43
Oct 31 18:32:24.373778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.373746    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"]
Oct 31 18:32:24.379301 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.379281    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.393435 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.393407    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"]
Oct 31 18:32:24.437685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.437652    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-88085e-openshift-default-dockercfg-96c8d\""
Oct 31 18:32:24.438881 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.438858    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"]
Oct 31 18:32:24.490794 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.490761    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:24.498760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.498736    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"]
Oct 31 18:32:24.517229 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.517198    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.517419 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.517252    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.517419 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.517364    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.517572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.517504    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.517572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.517541    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.517679 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.517652    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.517720 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.517692    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7b417c-a740-4b1d-ab49-467a568ef17f-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.517770 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.517721    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.517870 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.517852    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m2f7\" (UniqueName: \"kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-kube-api-access-4m2f7\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.619296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.619230    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.619743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.619325    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.619743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.619373    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.619743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.619407    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.619743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.619426    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.619743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.619467    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.619743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.619540    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7b417c-a740-4b1d-ab49-467a568ef17f-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.619743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.619580    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.619743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.619648    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m2f7\" (UniqueName: \"kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-kube-api-access-4m2f7\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.620300 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.620273    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.620423 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.620313    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.620813 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.620755    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7b417c-a740-4b1d-ab49-467a568ef17f-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.620971 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.620954    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.621043 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.620989    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.624505 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.624464    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.624833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.624814    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.678794 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.678762    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.678925 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.678817    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m2f7\" (UniqueName: \"kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-kube-api-access-4m2f7\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.692768 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.692739    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:24.766945 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.766907    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:24.930159 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.930061    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:24.931465 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.931435    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql" event={"ID":"2c6c9a2d-169b-46fc-9dea-bc46b6158003","Type":"ContainerStarted","Data":"3f24a335ff7bd4aeafd6940471ae5945023ef7e0392be422a9afe27a82aa0ca4"}
Oct 31 18:32:24.931626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:24.931497    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql" event={"ID":"2c6c9a2d-169b-46fc-9dea-bc46b6158003","Type":"ContainerStarted","Data":"868f1b2d932262e759e1b244b3ac392c11db5f56b2973d22c7ab2772c5b6df43"}
Oct 31 18:32:25.047193 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:25.047116    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql" podStartSLOduration=2.047099467 podStartE2EDuration="2.047099467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:25.045155856 +0000 UTC m=+691.191605718" watchObservedRunningTime="2026-10-31 18:32:25.047099467 +0000 UTC m=+691.193549361"
Oct 31 18:32:25.090204 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:25.090179    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"]
Oct 31 18:32:25.092430 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:32:25.092388    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7b417c_a740_4b1d_ab49_467a568ef17f.slice/crio-23967608848fdd7dc5aafb8daa4678a3bd87e5a52cd7d6fa5f21f1ed715bb9d0 WatchSource:0}: Error finding container 23967608848fdd7dc5aafb8daa4678a3bd87e5a52cd7d6fa5f21f1ed715bb9d0: Status 404 returned error can't find the container with id 23967608848fdd7dc5aafb8daa4678a3bd87e5a52cd7d6fa5f21f1ed715bb9d0
Oct 31 18:32:25.095267 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:25.095222    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:32:25.095388 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:25.095326    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:32:25.095438 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:25.095381    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:32:25.472017 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:32:25.471980    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:32:25.939511 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:25.939438    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" event={"ID":"be7b417c-a740-4b1d-ab49-467a568ef17f","Type":"ContainerStarted","Data":"384153cc820925dec13341644dcad1a6b1aef5c5042eedddc2801d5968a62096"}
Oct 31 18:32:25.940013 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:25.939564    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" event={"ID":"be7b417c-a740-4b1d-ab49-467a568ef17f","Type":"ContainerStarted","Data":"23967608848fdd7dc5aafb8daa4678a3bd87e5a52cd7d6fa5f21f1ed715bb9d0"}
Oct 31 18:32:25.941111 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:25.941082    4484 generic.go:356] "Generic (PLEG): container finished" podID="2c6c9a2d-169b-46fc-9dea-bc46b6158003" containerID="3f24a335ff7bd4aeafd6940471ae5945023ef7e0392be422a9afe27a82aa0ca4" exitCode=0
Oct 31 18:32:25.941247 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:25.941152    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql" event={"ID":"2c6c9a2d-169b-46fc-9dea-bc46b6158003","Type":"ContainerDied","Data":"3f24a335ff7bd4aeafd6940471ae5945023ef7e0392be422a9afe27a82aa0ca4"}
Oct 31 18:32:26.062666 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:26.062611    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" podStartSLOduration=2.062596889 podStartE2EDuration="2.062596889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:26.061605155 +0000 UTC m=+692.208055021" watchObservedRunningTime="2026-10-31 18:32:26.062596889 +0000 UTC m=+692.209046754"
Oct 31 18:32:26.693232 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:26.693192    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:26.698750 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:26.698719    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:26.946655 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:26.946546    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:26.947663 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:26.947642    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:32:27.098769 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.098738    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:27.204065 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.203968    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:27.245275 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.245234    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-xks-ca\") pod \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\" (UID: \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\") "
Oct 31 18:32:27.245504 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.245306    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c6c9a2d-169b-46fc-9dea-bc46b6158003-kube-api-access-pp9sm\" (UniqueName: \"kubernetes.io/projected/2c6c9a2d-169b-46fc-9dea-bc46b6158003-kube-api-access-pp9sm\") pod \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\" (UID: \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\") "
Oct 31 18:32:27.245504 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.245397    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-openshift-ca\") pod \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\" (UID: \"2c6c9a2d-169b-46fc-9dea-bc46b6158003\") "
Oct 31 18:32:27.245785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.245759    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-openshift-ca" pod "2c6c9a2d-169b-46fc-9dea-bc46b6158003" (UID: "2c6c9a2d-169b-46fc-9dea-bc46b6158003"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:27.247617 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.247588    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-xks-ca" pod "2c6c9a2d-169b-46fc-9dea-bc46b6158003" (UID: "2c6c9a2d-169b-46fc-9dea-bc46b6158003"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:32:27.247724 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.247637    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6c9a2d-169b-46fc-9dea-bc46b6158003-kube-api-access-pp9sm" pod "2c6c9a2d-169b-46fc-9dea-bc46b6158003" (UID: "2c6c9a2d-169b-46fc-9dea-bc46b6158003"). InnerVolumeSpecName "kube-api-access-pp9sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:27.346513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.346461    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:27.346513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.346511    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c6c9a2d-169b-46fc-9dea-bc46b6158003-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:27.346513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.346522    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp9sm\" (UniqueName: \"kubernetes.io/projected/2c6c9a2d-169b-46fc-9dea-bc46b6158003-kube-api-access-pp9sm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:27.366174 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.366141    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:27.525583 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.525545    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:27.799409 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.799323    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:32:27.952902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.952870    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"
Oct 31 18:32:27.953378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.952911    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql" event={"ID":"2c6c9a2d-169b-46fc-9dea-bc46b6158003","Type":"ContainerDied","Data":"868f1b2d932262e759e1b244b3ac392c11db5f56b2973d22c7ab2772c5b6df43"}
Oct 31 18:32:27.953378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:27.952970    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="868f1b2d932262e759e1b244b3ac392c11db5f56b2973d22c7ab2772c5b6df43"
Oct 31 18:32:28.346716 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.346651    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:28.352764 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.352742    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:32:28.458115 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458082    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-data\") pod \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") "
Oct 31 18:32:28.458115 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458120    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istiod-ca-cert\") pod \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") "
Oct 31 18:32:28.458368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458141    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-kube-api-access-lznrc\" (UniqueName: \"kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-kube-api-access-lznrc\") pod \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") "
Oct 31 18:32:28.458368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458188    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-certs\") pod \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") "
Oct 31 18:32:28.458368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458283    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-token\" (UniqueName: \"kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-token\") pod \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") "
Oct 31 18:32:28.458368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458319    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-podinfo\") pod \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") "
Oct 31 18:32:28.458368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458354    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-credential-socket\") pod \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") "
Oct 31 18:32:28.458669 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458391    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-envoy\") pod \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") "
Oct 31 18:32:28.458669 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458410    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-data" pod "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" (UID: "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:28.458669 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458515    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-socket\") pod \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\" (UID: \"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71\") "
Oct 31 18:32:28.458669 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458649    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-credential-socket" pod "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" (UID: "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:28.458863 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458670    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istiod-ca-cert" pod "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" (UID: "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:28.458919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458871    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-socket" pod "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" (UID: "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:28.458919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.458906    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-certs" pod "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" (UID: "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:28.459223 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.459175    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:28.459282 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.459230    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:28.459282 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.459268    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:28.459350 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.459284    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:28.459350 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.459297    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:28.460735 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.460715    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-kube-api-access-lznrc" pod "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" (UID: "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71"). InnerVolumeSpecName "kube-api-access-lznrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:28.460847 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.460827    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-podinfo" pod "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" (UID: "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:32:28.461142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.461117    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-envoy" pod "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" (UID: "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:32:28.461142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.461136    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-token" pod "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" (UID: "4fe4dff5-13c8-486a-8c2f-80d8cc40fe71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:28.560352 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.560319    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lznrc\" (UniqueName: \"kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-kube-api-access-lznrc\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:28.560352 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.560348    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:28.560352 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.560359    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:28.560623 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.560368    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:28.730398 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.730311    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:28.959820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.959790    4484 generic.go:356] "Generic (PLEG): container finished" podID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerID="ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343" exitCode=0
Oct 31 18:32:28.960243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.959848    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"
Oct 31 18:32:28.960243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.959872    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" event={"ID":"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71","Type":"ContainerDied","Data":"ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343"}
Oct 31 18:32:28.960243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.959939    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt" event={"ID":"4fe4dff5-13c8-486a-8c2f-80d8cc40fe71","Type":"ContainerDied","Data":"9163ec058ae8313733e76747fa4bcdff881b153f6b008565e70586573c99d704"}
Oct 31 18:32:28.960243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.959972    4484 scope.go:122] "RemoveContainer" containerID="ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343"
Oct 31 18:32:28.970701 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.970679    4484 scope.go:122] "RemoveContainer" containerID="596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563"
Oct 31 18:32:28.979554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.979529    4484 scope.go:122] "RemoveContainer" containerID="ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343"
Oct 31 18:32:28.979849 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:32:28.979829    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343\": container with ID starting with ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343 not found: ID does not exist" containerID="ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343"
Oct 31 18:32:28.979902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.979860    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343"} err="failed to get container status \"ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343\": rpc error: code = NotFound desc = could not find container \"ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343\": container with ID starting with ba5df9036159dd639e74fe018b6fca39fe4c2d005382d6556e486c5c7ab88343 not found: ID does not exist"
Oct 31 18:32:28.979902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.979877    4484 scope.go:122] "RemoveContainer" containerID="596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563"
Oct 31 18:32:28.980098 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:32:28.980078    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563\": container with ID starting with 596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563 not found: ID does not exist" containerID="596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563"
Oct 31 18:32:28.980138 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:28.980108    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563"} err="failed to get container status \"596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563\": rpc error: code = NotFound desc = could not find container \"596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563\": container with ID starting with 596ed71dd09de22f1e66f6a80be83a384087555fc55e98cbbe90e1aa3a252563 not found: ID does not exist"
Oct 31 18:32:29.096816 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.096781    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:29.132158 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.132128    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"]
Oct 31 18:32:29.183566 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.183531    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e4c133ca-openshift-default-548c66c5f-8qnmt"]
Oct 31 18:32:29.394239 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.394137    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f" podUID="a10b847c-1752-49bd-b16d-6f4f2a3a29f1" containerName="payload-processing" containerID="cri-o://1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d" gracePeriod=25
Oct 31 18:32:29.499917 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.499873    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d" podUID="1a356ee7-2002-47b0-8199-531af545cf1b" containerName="payload-pre-processing" containerID="cri-o://98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd" gracePeriod=25
Oct 31 18:32:29.661313 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.661241    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"
Oct 31 18:32:29.726814 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.726779    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:29.769193 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.769166    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"
Oct 31 18:32:29.772633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.772609    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-plugins-config-volume\") pod \"a10b847c-1752-49bd-b16d-6f4f2a3a29f1\" (UID: \"a10b847c-1752-49bd-b16d-6f4f2a3a29f1\") "
Oct 31 18:32:29.772770 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.772648    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-kube-api-access-pq7qt\" (UniqueName: \"kubernetes.io/projected/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-kube-api-access-pq7qt\") pod \"a10b847c-1752-49bd-b16d-6f4f2a3a29f1\" (UID: \"a10b847c-1752-49bd-b16d-6f4f2a3a29f1\") "
Oct 31 18:32:29.772946 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.772922    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-plugins-config-volume" pod "a10b847c-1752-49bd-b16d-6f4f2a3a29f1" (UID: "a10b847c-1752-49bd-b16d-6f4f2a3a29f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:29.774978 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.774955    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-kube-api-access-pq7qt" pod "a10b847c-1752-49bd-b16d-6f4f2a3a29f1" (UID: "a10b847c-1752-49bd-b16d-6f4f2a3a29f1"). InnerVolumeSpecName "kube-api-access-pq7qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:29.873670 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.873635    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a356ee7-2002-47b0-8199-531af545cf1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a356ee7-2002-47b0-8199-531af545cf1b-plugins-config-volume\") pod \"1a356ee7-2002-47b0-8199-531af545cf1b\" (UID: \"1a356ee7-2002-47b0-8199-531af545cf1b\") "
Oct 31 18:32:29.873857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.873825    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a356ee7-2002-47b0-8199-531af545cf1b-kube-api-access-c8dlk\" (UniqueName: \"kubernetes.io/projected/1a356ee7-2002-47b0-8199-531af545cf1b-kube-api-access-c8dlk\") pod \"1a356ee7-2002-47b0-8199-531af545cf1b\" (UID: \"1a356ee7-2002-47b0-8199-531af545cf1b\") "
Oct 31 18:32:29.874024 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.874002    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a356ee7-2002-47b0-8199-531af545cf1b-plugins-config-volume" pod "1a356ee7-2002-47b0-8199-531af545cf1b" (UID: "1a356ee7-2002-47b0-8199-531af545cf1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:29.874126 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.874107    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:29.874162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.874131    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq7qt\" (UniqueName: \"kubernetes.io/projected/a10b847c-1752-49bd-b16d-6f4f2a3a29f1-kube-api-access-pq7qt\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:29.874162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.874143    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a356ee7-2002-47b0-8199-531af545cf1b-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:29.876243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.876218    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a356ee7-2002-47b0-8199-531af545cf1b-kube-api-access-c8dlk" pod "1a356ee7-2002-47b0-8199-531af545cf1b" (UID: "1a356ee7-2002-47b0-8199-531af545cf1b"). InnerVolumeSpecName "kube-api-access-c8dlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:29.946876 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.946787    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:29.966935 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.966900    4484 generic.go:356] "Generic (PLEG): container finished" podID="1a356ee7-2002-47b0-8199-531af545cf1b" containerID="98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd" exitCode=0
Oct 31 18:32:29.967357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.966951    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"
Oct 31 18:32:29.967357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.966974    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d" event={"ID":"1a356ee7-2002-47b0-8199-531af545cf1b","Type":"ContainerDied","Data":"98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd"}
Oct 31 18:32:29.967357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.967052    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d" event={"ID":"1a356ee7-2002-47b0-8199-531af545cf1b","Type":"ContainerDied","Data":"d2da2b0fb90bb1485882961289b6bc771e23d7181f5e7fd37f4a060539ee4183"}
Oct 31 18:32:29.967357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.967080    4484 scope.go:122] "RemoveContainer" containerID="98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd"
Oct 31 18:32:29.968632 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.968596    4484 generic.go:356] "Generic (PLEG): container finished" podID="a10b847c-1752-49bd-b16d-6f4f2a3a29f1" containerID="1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d" exitCode=0
Oct 31 18:32:29.968754 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.968688    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f" event={"ID":"a10b847c-1752-49bd-b16d-6f4f2a3a29f1","Type":"ContainerDied","Data":"1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d"}
Oct 31 18:32:29.968754 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.968749    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f" event={"ID":"a10b847c-1752-49bd-b16d-6f4f2a3a29f1","Type":"ContainerDied","Data":"af1d776bd5b7db1abd57c09473c6a8ecb8ed3f6a58affa17137da423734f8781"}
Oct 31 18:32:29.968845 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.968816    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"
Oct 31 18:32:29.974902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.974870    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8dlk\" (UniqueName: \"kubernetes.io/projected/1a356ee7-2002-47b0-8199-531af545cf1b-kube-api-access-c8dlk\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:29.978892 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.978869    4484 scope.go:122] "RemoveContainer" containerID="98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd"
Oct 31 18:32:29.979378 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:32:29.979225    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd\": container with ID starting with 98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd not found: ID does not exist" containerID="98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd"
Oct 31 18:32:29.979378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.979251    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd"} err="failed to get container status \"98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd\": rpc error: code = NotFound desc = could not find container \"98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd\": container with ID starting with 98c9bdcb150c48931f507594d19c5ca15218b02eda458aa933b9898bbd456ffd not found: ID does not exist"
Oct 31 18:32:29.979378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.979268    4484 scope.go:122] "RemoveContainer" containerID="1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d"
Oct 31 18:32:29.988682 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.988653    4484 scope.go:122] "RemoveContainer" containerID="1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d"
Oct 31 18:32:29.988945 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:32:29.988925    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d\": container with ID starting with 1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d not found: ID does not exist" containerID="1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d"
Oct 31 18:32:29.989020 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.988955    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d"} err="failed to get container status \"1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d\": rpc error: code = NotFound desc = could not find container \"1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d\": container with ID starting with 1ac4a1ac7ba31fc230f14f05962d5e0ce2c8890cb28bfecf4fb99d8cee4d515d not found: ID does not exist"
Oct 31 18:32:29.994069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:29.994043    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" path="/var/lib/kubelet/pods/4fe4dff5-13c8-486a-8c2f-80d8cc40fe71/volumes"
Oct 31 18:32:30.133708 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.133674    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"]
Oct 31 18:32:30.139419 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.139392    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bbf5caaf-68b4b55967-fj85d"]
Oct 31 18:32:30.202887 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.202765    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:30.251949 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.251912    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"]
Oct 31 18:32:30.256772 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.256744    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-bbf5caaf-6c4bdff4bc-m7m9f"]
Oct 31 18:32:30.306366 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.306332    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"]
Oct 31 18:32:30.307040 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307022    4484 cpu_manager.go:414] "containerMap: removing container" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerName="istio-proxy"
Oct 31 18:32:30.307098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307042    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerName="istio-proxy"
Oct 31 18:32:30.307098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307067    4484 cpu_manager.go:414] "containerMap: removing container" podUID="2c6c9a2d-169b-46fc-9dea-bc46b6158003" containerName="revoke-keys"
Oct 31 18:32:30.307098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307077    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c6c9a2d-169b-46fc-9dea-bc46b6158003" containerName="revoke-keys"
Oct 31 18:32:30.307191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307104    4484 cpu_manager.go:414] "containerMap: removing container" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerName="istio-proxy"
Oct 31 18:32:30.307191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307111    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerName="istio-proxy"
Oct 31 18:32:30.307191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307123    4484 cpu_manager.go:414] "containerMap: removing container" podUID="1a356ee7-2002-47b0-8199-531af545cf1b" containerName="payload-pre-processing"
Oct 31 18:32:30.307191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307131    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a356ee7-2002-47b0-8199-531af545cf1b" containerName="payload-pre-processing"
Oct 31 18:32:30.307191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307154    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a10b847c-1752-49bd-b16d-6f4f2a3a29f1" containerName="payload-processing"
Oct 31 18:32:30.307191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307162    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10b847c-1752-49bd-b16d-6f4f2a3a29f1" containerName="payload-processing"
Oct 31 18:32:30.307387 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307244    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a356ee7-2002-47b0-8199-531af545cf1b" containerName="payload-pre-processing"
Oct 31 18:32:30.307387 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307256    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerName="istio-proxy"
Oct 31 18:32:30.307387 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307267    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10b847c-1752-49bd-b16d-6f4f2a3a29f1" containerName="payload-processing"
Oct 31 18:32:30.307387 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307277    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe4dff5-13c8-486a-8c2f-80d8cc40fe71" containerName="istio-proxy"
Oct 31 18:32:30.307387 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.307288    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c6c9a2d-169b-46fc-9dea-bc46b6158003" containerName="revoke-keys"
Oct 31 18:32:30.312692 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.312666    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:30.366722 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.366691    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"]
Oct 31 18:32:30.433205 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.433173    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:32:30.433205 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.433177    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:32:30.480078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.479983    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcwx2\" (UniqueName: \"kubernetes.io/projected/d51e09a2-b0a3-40d9-889f-cab103a056ef-kube-api-access-hcwx2\") pod \"maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc\" (UID: \"d51e09a2-b0a3-40d9-889f-cab103a056ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:30.480256 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.480105    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc\" (UID: \"d51e09a2-b0a3-40d9-889f-cab103a056ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:30.480256 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.480143    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc\" (UID: \"d51e09a2-b0a3-40d9-889f-cab103a056ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:30.580816 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.580773    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc\" (UID: \"d51e09a2-b0a3-40d9-889f-cab103a056ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:30.580998 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.580827    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc\" (UID: \"d51e09a2-b0a3-40d9-889f-cab103a056ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:30.580998 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.580896    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcwx2\" (UniqueName: \"kubernetes.io/projected/d51e09a2-b0a3-40d9-889f-cab103a056ef-kube-api-access-hcwx2\") pod \"maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc\" (UID: \"d51e09a2-b0a3-40d9-889f-cab103a056ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:30.581587 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.581565    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc\" (UID: \"d51e09a2-b0a3-40d9-889f-cab103a056ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:30.583374 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.583349    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc\" (UID: \"d51e09a2-b0a3-40d9-889f-cab103a056ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:30.638560 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.638520    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcwx2\" (UniqueName: \"kubernetes.io/projected/d51e09a2-b0a3-40d9-889f-cab103a056ef-kube-api-access-hcwx2\") pod \"maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc\" (UID: \"d51e09a2-b0a3-40d9-889f-cab103a056ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:30.773310 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.773277    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:30.926404 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:30.926368    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:31.326778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:31.326749    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"]
Oct 31 18:32:31.328219 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:32:31.328184    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51e09a2_b0a3_40d9_889f_cab103a056ef.slice/crio-4692ee1b2fef70339c59530a0a2139e5de5524b732f031c78c296cd811d08117 WatchSource:0}: Error finding container 4692ee1b2fef70339c59530a0a2139e5de5524b732f031c78c296cd811d08117: Status 404 returned error can't find the container with id 4692ee1b2fef70339c59530a0a2139e5de5524b732f031c78c296cd811d08117
Oct 31 18:32:31.581752 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:31.581663    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:31.991898 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:31.991871    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a356ee7-2002-47b0-8199-531af545cf1b" path="/var/lib/kubelet/pods/1a356ee7-2002-47b0-8199-531af545cf1b/volumes"
Oct 31 18:32:31.992252 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:31.992240    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10b847c-1752-49bd-b16d-6f4f2a3a29f1" path="/var/lib/kubelet/pods/a10b847c-1752-49bd-b16d-6f4f2a3a29f1/volumes"
Oct 31 18:32:31.992540 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:31.992520    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc" event={"ID":"d51e09a2-b0a3-40d9-889f-cab103a056ef","Type":"ContainerStarted","Data":"2b9177016cd950103e4c06584be4edf3873d088c197eec67264af084fddce1f3"}
Oct 31 18:32:31.992590 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:31.992568    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc" event={"ID":"d51e09a2-b0a3-40d9-889f-cab103a056ef","Type":"ContainerStarted","Data":"4692ee1b2fef70339c59530a0a2139e5de5524b732f031c78c296cd811d08117"}
Oct 31 18:32:32.186194 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:32.186141    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc" podStartSLOduration=2.186125765 podStartE2EDuration="2.186125765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:32.184748982 +0000 UTC m=+698.331198848" watchObservedRunningTime="2026-10-31 18:32:32.186125765 +0000 UTC m=+698.332575630"
Oct 31 18:32:32.232574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:32.232529    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:32.399174 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:32.399081    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:32.993703 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:32.993662    4484 generic.go:356] "Generic (PLEG): container finished" podID="d51e09a2-b0a3-40d9-889f-cab103a056ef" containerID="2b9177016cd950103e4c06584be4edf3873d088c197eec67264af084fddce1f3" exitCode=0
Oct 31 18:32:32.993889 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:32.993747    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc" event={"ID":"d51e09a2-b0a3-40d9-889f-cab103a056ef","Type":"ContainerDied","Data":"2b9177016cd950103e4c06584be4edf3873d088c197eec67264af084fddce1f3"}
Oct 31 18:32:33.240396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.240353    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:33.448492 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.448384    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:33.638789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.638752    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"]
Oct 31 18:32:33.642448 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.642430    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"
Oct 31 18:32:33.697259 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.697203    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-88085e-dockercfg-5mxnm\""
Oct 31 18:32:33.700652 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.700557    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"]
Oct 31 18:32:33.702053 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.702017    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-88085e\""
Oct 31 18:32:33.776129 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.776095    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"]
Oct 31 18:32:33.779923 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.779902    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"
Oct 31 18:32:33.814074 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.814043    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f711e6a-d69b-4211-a01d-ba243b48501e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp\" (UID: \"5f711e6a-d69b-4211-a01d-ba243b48501e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"
Oct 31 18:32:33.814236 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.814221    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6dvn\" (UniqueName: \"kubernetes.io/projected/5f711e6a-d69b-4211-a01d-ba243b48501e-kube-api-access-f6dvn\") pod \"payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp\" (UID: \"5f711e6a-d69b-4211-a01d-ba243b48501e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"
Oct 31 18:32:33.836781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.836753    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"]
Oct 31 18:32:33.915042 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.914994    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6dvn\" (UniqueName: \"kubernetes.io/projected/5f711e6a-d69b-4211-a01d-ba243b48501e-kube-api-access-f6dvn\") pod \"payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp\" (UID: \"5f711e6a-d69b-4211-a01d-ba243b48501e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"
Oct 31 18:32:33.915246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.915060    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f711e6a-d69b-4211-a01d-ba243b48501e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp\" (UID: \"5f711e6a-d69b-4211-a01d-ba243b48501e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"
Oct 31 18:32:33.915246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.915106    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76\" (UID: \"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"
Oct 31 18:32:33.915246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.915169    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5hf5\" (UniqueName: \"kubernetes.io/projected/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-kube-api-access-h5hf5\") pod \"payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76\" (UID: \"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"
Oct 31 18:32:33.915853 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.915827    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f711e6a-d69b-4211-a01d-ba243b48501e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp\" (UID: \"5f711e6a-d69b-4211-a01d-ba243b48501e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"
Oct 31 18:32:33.972730 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:33.972660    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6dvn\" (UniqueName: \"kubernetes.io/projected/5f711e6a-d69b-4211-a01d-ba243b48501e-kube-api-access-f6dvn\") pod \"payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp\" (UID: \"5f711e6a-d69b-4211-a01d-ba243b48501e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"
Oct 31 18:32:34.016172 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.016137    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76\" (UID: \"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"
Oct 31 18:32:34.016348 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.016202    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5hf5\" (UniqueName: \"kubernetes.io/projected/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-kube-api-access-h5hf5\") pod \"payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76\" (UID: \"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"
Oct 31 18:32:34.017129 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.017104    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76\" (UID: \"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"
Oct 31 18:32:34.074232 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.074201    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5hf5\" (UniqueName: \"kubernetes.io/projected/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-kube-api-access-h5hf5\") pod \"payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76\" (UID: \"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"
Oct 31 18:32:34.093946 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.093902    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"
Oct 31 18:32:34.144172 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.144139    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:34.255867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.255838    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"
Oct 31 18:32:34.319543 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.319456    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-xks-ca\") pod \"d51e09a2-b0a3-40d9-889f-cab103a056ef\" (UID: \"d51e09a2-b0a3-40d9-889f-cab103a056ef\") "
Oct 31 18:32:34.319824 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.319800    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d51e09a2-b0a3-40d9-889f-cab103a056ef-kube-api-access-hcwx2\" (UniqueName: \"kubernetes.io/projected/d51e09a2-b0a3-40d9-889f-cab103a056ef-kube-api-access-hcwx2\") pod \"d51e09a2-b0a3-40d9-889f-cab103a056ef\" (UID: \"d51e09a2-b0a3-40d9-889f-cab103a056ef\") "
Oct 31 18:32:34.320206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.320043    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-openshift-ca\") pod \"d51e09a2-b0a3-40d9-889f-cab103a056ef\" (UID: \"d51e09a2-b0a3-40d9-889f-cab103a056ef\") "
Oct 31 18:32:34.320439 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.320412    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-openshift-ca" pod "d51e09a2-b0a3-40d9-889f-cab103a056ef" (UID: "d51e09a2-b0a3-40d9-889f-cab103a056ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:34.322962 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.322935    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:34.323080 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.323057    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d51e09a2-b0a3-40d9-889f-cab103a056ef-kube-api-access-hcwx2" pod "d51e09a2-b0a3-40d9-889f-cab103a056ef" (UID: "d51e09a2-b0a3-40d9-889f-cab103a056ef"). InnerVolumeSpecName "kube-api-access-hcwx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:34.323080 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.323065    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-xks-ca" pod "d51e09a2-b0a3-40d9-889f-cab103a056ef" (UID: "d51e09a2-b0a3-40d9-889f-cab103a056ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:32:34.423896 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.423860    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d51e09a2-b0a3-40d9-889f-cab103a056ef-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:34.423896 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.423894    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcwx2\" (UniqueName: \"kubernetes.io/projected/d51e09a2-b0a3-40d9-889f-cab103a056ef-kube-api-access-hcwx2\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:34.474269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.474244    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"]
Oct 31 18:32:34.475763 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:32:34.475741    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cd62f3_4c84_4f3b_a0a8_39d9b1acde99.slice/crio-2b1464619e0bd30cd3834eadf0ed6a7d8ba8c5c7c36ce2d38eab6e88528932cc WatchSource:0}: Error finding container 2b1464619e0bd30cd3834eadf0ed6a7d8ba8c5c7c36ce2d38eab6e88528932cc: Status 404 returned error can't find the container with id 2b1464619e0bd30cd3834eadf0ed6a7d8ba8c5c7c36ce2d38eab6e88528932cc
Oct 31 18:32:34.642661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.642640    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"]
Oct 31 18:32:34.644918 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:32:34.644891    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f711e6a_d69b_4211_a01d_ba243b48501e.slice/crio-aa364212eda2d3a14cdc80dd086b8844709ca8c6ae178e0f1851577127ea96d0 WatchSource:0}: Error finding container aa364212eda2d3a14cdc80dd086b8844709ca8c6ae178e0f1851577127ea96d0: Status 404 returned error can't find the container with id aa364212eda2d3a14cdc80dd086b8844709ca8c6ae178e0f1851577127ea96d0
Oct 31 18:32:34.752018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:34.751978    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:35.004603 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.004573    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"
Oct 31 18:32:35.004771 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.004572    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc" event={"ID":"d51e09a2-b0a3-40d9-889f-cab103a056ef","Type":"ContainerDied","Data":"4692ee1b2fef70339c59530a0a2139e5de5524b732f031c78c296cd811d08117"}
Oct 31 18:32:35.004771 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.004707    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4692ee1b2fef70339c59530a0a2139e5de5524b732f031c78c296cd811d08117"
Oct 31 18:32:35.006209 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.006183    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76" event={"ID":"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99","Type":"ContainerStarted","Data":"406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b"}
Oct 31 18:32:35.006354 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.006238    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"
Oct 31 18:32:35.006354 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.006260    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76" event={"ID":"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99","Type":"ContainerStarted","Data":"2b1464619e0bd30cd3834eadf0ed6a7d8ba8c5c7c36ce2d38eab6e88528932cc"}
Oct 31 18:32:35.007757 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.007734    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp" event={"ID":"5f711e6a-d69b-4211-a01d-ba243b48501e","Type":"ContainerStarted","Data":"e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b"}
Oct 31 18:32:35.007857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.007779    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp" event={"ID":"5f711e6a-d69b-4211-a01d-ba243b48501e","Type":"ContainerStarted","Data":"aa364212eda2d3a14cdc80dd086b8844709ca8c6ae178e0f1851577127ea96d0"}
Oct 31 18:32:35.007857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.007816    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"
Oct 31 18:32:35.123223 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.123169    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76" podStartSLOduration=2.123155401 podStartE2EDuration="2.123155401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:35.122009778 +0000 UTC m=+701.268459644" watchObservedRunningTime="2026-10-31 18:32:35.123155401 +0000 UTC m=+701.269605265"
Oct 31 18:32:35.239453 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.239396    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp" podStartSLOduration=2.239384026 podStartE2EDuration="2.239384026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:35.237400085 +0000 UTC m=+701.383849949" watchObservedRunningTime="2026-10-31 18:32:35.239384026 +0000 UTC m=+701.385833914"
Oct 31 18:32:35.352862 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.352828    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:35.550849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:35.550755    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:36.436109 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:36.436076    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:36.649270 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:36.649236    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:36.813702 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:36.813665    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:41.430330 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:41.430296    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"]
Oct 31 18:32:41.536908 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:41.536875    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"]
Oct 31 18:32:41.813128 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:41.813098    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:41.967751 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:41.967717    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:42.123081 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:42.122983    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:42.522455 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:42.522417    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:42.800321 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:42.800240    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:43.002606 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:43.002573    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"]
Oct 31 18:32:43.053404 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:43.053316    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c7a8371-kskql"]
Oct 31 18:32:43.125248 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:43.125211    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:43.924982 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:43.924941    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:43.993242 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:43.993207    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c6c9a2d-169b-46fc-9dea-bc46b6158003" path="/var/lib/kubelet/pods/2c6c9a2d-169b-46fc-9dea-bc46b6158003/volumes"
Oct 31 18:32:44.080519 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:44.080461    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:44.529123 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:44.529090    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:44.744988 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:44.744957    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:45.257206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.257176    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:45.433688 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.433655    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:45.792292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.792250    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"]
Oct 31 18:32:45.793261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.793236    4484 cpu_manager.go:414] "containerMap: removing container" podUID="d51e09a2-b0a3-40d9-889f-cab103a056ef" containerName="revoke-keys"
Oct 31 18:32:45.793261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.793262    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d51e09a2-b0a3-40d9-889f-cab103a056ef" containerName="revoke-keys"
Oct 31 18:32:45.793555 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.793541    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d51e09a2-b0a3-40d9-889f-cab103a056ef" containerName="revoke-keys"
Oct 31 18:32:45.797827 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.797799    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"
Oct 31 18:32:45.835930 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.835896    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnh2l\" (UniqueName: \"kubernetes.io/projected/461e65fc-44a3-437f-93a9-bccc4a4d45b4-kube-api-access-xnh2l\") pod \"payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk\" (UID: \"461e65fc-44a3-437f-93a9-bccc4a4d45b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"
Oct 31 18:32:45.836078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.836036    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461e65fc-44a3-437f-93a9-bccc4a4d45b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk\" (UID: \"461e65fc-44a3-437f-93a9-bccc4a4d45b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"
Oct 31 18:32:45.849526 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.849465    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"]
Oct 31 18:32:45.897688 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.897656    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-6279f5\""
Oct 31 18:32:45.897688 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.897657    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-6279f5-dockercfg-x5vh6\""
Oct 31 18:32:45.936631 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.936592    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnh2l\" (UniqueName: \"kubernetes.io/projected/461e65fc-44a3-437f-93a9-bccc4a4d45b4-kube-api-access-xnh2l\") pod \"payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk\" (UID: \"461e65fc-44a3-437f-93a9-bccc4a4d45b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"
Oct 31 18:32:45.936815 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.936685    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461e65fc-44a3-437f-93a9-bccc4a4d45b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk\" (UID: \"461e65fc-44a3-437f-93a9-bccc4a4d45b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"
Oct 31 18:32:45.937373 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.937352    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461e65fc-44a3-437f-93a9-bccc4a4d45b4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk\" (UID: \"461e65fc-44a3-437f-93a9-bccc4a4d45b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"
Oct 31 18:32:45.994275 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:45.994244    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnh2l\" (UniqueName: \"kubernetes.io/projected/461e65fc-44a3-437f-93a9-bccc4a4d45b4-kube-api-access-xnh2l\") pod \"payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk\" (UID: \"461e65fc-44a3-437f-93a9-bccc4a4d45b4\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"
Oct 31 18:32:46.026253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.026225    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"
Oct 31 18:32:46.026437 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.026294    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"
Oct 31 18:32:46.110877 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.110792    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"
Oct 31 18:32:46.431367 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.431262    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg" podUID="13097b93-539f-4187-86b4-10b649a9ef85" containerName="payload-processing" containerID="cri-o://e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a" gracePeriod=25
Oct 31 18:32:46.490225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.490200    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"]
Oct 31 18:32:46.522430 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:32:46.522401    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461e65fc_44a3_437f_93a9_bccc4a4d45b4.slice/crio-34d1d8d79017d9de35ab3cb7bf4ec615d11141a4226ed752ef6c1eb0ffab07f6 WatchSource:0}: Error finding container 34d1d8d79017d9de35ab3cb7bf4ec615d11141a4226ed752ef6c1eb0ffab07f6: Status 404 returned error can't find the container with id 34d1d8d79017d9de35ab3cb7bf4ec615d11141a4226ed752ef6c1eb0ffab07f6
Oct 31 18:32:46.537942 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.537908    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg" podUID="9a09f2c6-e58f-436b-a794-63b0da9037fb" containerName="payload-pre-processing" containerID="cri-o://a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86" gracePeriod=25
Oct 31 18:32:46.679794 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.679769    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"
Oct 31 18:32:46.748959 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.748910    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13097b93-539f-4187-86b4-10b649a9ef85-kube-api-access-2bpp4\" (UniqueName: \"kubernetes.io/projected/13097b93-539f-4187-86b4-10b649a9ef85-kube-api-access-2bpp4\") pod \"13097b93-539f-4187-86b4-10b649a9ef85\" (UID: \"13097b93-539f-4187-86b4-10b649a9ef85\") "
Oct 31 18:32:46.749195 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.749017    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13097b93-539f-4187-86b4-10b649a9ef85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13097b93-539f-4187-86b4-10b649a9ef85-plugins-config-volume\") pod \"13097b93-539f-4187-86b4-10b649a9ef85\" (UID: \"13097b93-539f-4187-86b4-10b649a9ef85\") "
Oct 31 18:32:46.749508 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.749461    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13097b93-539f-4187-86b4-10b649a9ef85-plugins-config-volume" pod "13097b93-539f-4187-86b4-10b649a9ef85" (UID: "13097b93-539f-4187-86b4-10b649a9ef85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:46.751292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.751262    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13097b93-539f-4187-86b4-10b649a9ef85-kube-api-access-2bpp4" pod "13097b93-539f-4187-86b4-10b649a9ef85" (UID: "13097b93-539f-4187-86b4-10b649a9ef85"). InnerVolumeSpecName "kube-api-access-2bpp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:46.788554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.788527    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"
Oct 31 18:32:46.850235 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.850201    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a09f2c6-e58f-436b-a794-63b0da9037fb-kube-api-access-wd7fv\" (UniqueName: \"kubernetes.io/projected/9a09f2c6-e58f-436b-a794-63b0da9037fb-kube-api-access-wd7fv\") pod \"9a09f2c6-e58f-436b-a794-63b0da9037fb\" (UID: \"9a09f2c6-e58f-436b-a794-63b0da9037fb\") "
Oct 31 18:32:46.850418 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.850309    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a09f2c6-e58f-436b-a794-63b0da9037fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a09f2c6-e58f-436b-a794-63b0da9037fb-plugins-config-volume\") pod \"9a09f2c6-e58f-436b-a794-63b0da9037fb\" (UID: \"9a09f2c6-e58f-436b-a794-63b0da9037fb\") "
Oct 31 18:32:46.850522 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.850512    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13097b93-539f-4187-86b4-10b649a9ef85-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:46.850582 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.850525    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bpp4\" (UniqueName: \"kubernetes.io/projected/13097b93-539f-4187-86b4-10b649a9ef85-kube-api-access-2bpp4\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:46.850756 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.850732    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a09f2c6-e58f-436b-a794-63b0da9037fb-plugins-config-volume" pod "9a09f2c6-e58f-436b-a794-63b0da9037fb" (UID: "9a09f2c6-e58f-436b-a794-63b0da9037fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:46.852407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.852386    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a09f2c6-e58f-436b-a794-63b0da9037fb-kube-api-access-wd7fv" pod "9a09f2c6-e58f-436b-a794-63b0da9037fb" (UID: "9a09f2c6-e58f-436b-a794-63b0da9037fb"). InnerVolumeSpecName "kube-api-access-wd7fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:46.951944 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.951865    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd7fv\" (UniqueName: \"kubernetes.io/projected/9a09f2c6-e58f-436b-a794-63b0da9037fb-kube-api-access-wd7fv\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:46.951944 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:46.951893    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a09f2c6-e58f-436b-a794-63b0da9037fb-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:47.073141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.073108    4484 generic.go:356] "Generic (PLEG): container finished" podID="9a09f2c6-e58f-436b-a794-63b0da9037fb" containerID="a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86" exitCode=0
Oct 31 18:32:47.073325 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.073153    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"
Oct 31 18:32:47.073325 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.073190    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg" event={"ID":"9a09f2c6-e58f-436b-a794-63b0da9037fb","Type":"ContainerDied","Data":"a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86"}
Oct 31 18:32:47.073325 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.073253    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg" event={"ID":"9a09f2c6-e58f-436b-a794-63b0da9037fb","Type":"ContainerDied","Data":"4d647be6f3da187a0dd2e550db0d9228541c816499027b1f8b6d829f6ac63b23"}
Oct 31 18:32:47.073325 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.073276    4484 scope.go:122] "RemoveContainer" containerID="a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86"
Oct 31 18:32:47.074660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.074641    4484 generic.go:356] "Generic (PLEG): container finished" podID="13097b93-539f-4187-86b4-10b649a9ef85" containerID="e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a" exitCode=0
Oct 31 18:32:47.074780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.074680    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"
Oct 31 18:32:47.074780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.074713    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg" event={"ID":"13097b93-539f-4187-86b4-10b649a9ef85","Type":"ContainerDied","Data":"e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a"}
Oct 31 18:32:47.074780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.074775    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg" event={"ID":"13097b93-539f-4187-86b4-10b649a9ef85","Type":"ContainerDied","Data":"73472ccf66428849a047c9163d63f6ab3fc139890d415cf0bc9aec8b1117b799"}
Oct 31 18:32:47.076528 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.076501    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk" event={"ID":"461e65fc-44a3-437f-93a9-bccc4a4d45b4","Type":"ContainerStarted","Data":"ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc"}
Oct 31 18:32:47.076633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.076562    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"
Oct 31 18:32:47.076633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.076584    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk" event={"ID":"461e65fc-44a3-437f-93a9-bccc4a4d45b4","Type":"ContainerStarted","Data":"34d1d8d79017d9de35ab3cb7bf4ec615d11141a4226ed752ef6c1eb0ffab07f6"}
Oct 31 18:32:47.083085 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.083062    4484 scope.go:122] "RemoveContainer" containerID="a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86"
Oct 31 18:32:47.083369 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:32:47.083351    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86\": container with ID starting with a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86 not found: ID does not exist" containerID="a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86"
Oct 31 18:32:47.083456 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.083376    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86"} err="failed to get container status \"a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86\": rpc error: code = NotFound desc = could not find container \"a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86\": container with ID starting with a63209e7d7374886cd56f5b402ab2693a4a766a82494925f5909fb29cd463d86 not found: ID does not exist"
Oct 31 18:32:47.083456 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.083395    4484 scope.go:122] "RemoveContainer" containerID="e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a"
Oct 31 18:32:47.091750 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.091728    4484 scope.go:122] "RemoveContainer" containerID="e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a"
Oct 31 18:32:47.092043 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:32:47.092025    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a\": container with ID starting with e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a not found: ID does not exist" containerID="e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a"
Oct 31 18:32:47.092043 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.092051    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a"} err="failed to get container status \"e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a\": rpc error: code = NotFound desc = could not find container \"e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a\": container with ID starting with e9811fd6381f0b1c55079af2253abcd4baf7541209524a64745d3bbc359c9f3a not found: ID does not exist"
Oct 31 18:32:47.162499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.149756    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:32:47.257743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.257690    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk" podStartSLOduration=2.257674209 podStartE2EDuration="2.257674209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:47.25597792 +0000 UTC m=+713.402427785" watchObservedRunningTime="2026-10-31 18:32:47.257674209 +0000 UTC m=+713.404124074"
Oct 31 18:32:47.389840 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.389808    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"]
Oct 31 18:32:47.407069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.407021    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8d3cca1f-769cffc655-747jg"]
Oct 31 18:32:47.529114 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.529022    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"]
Oct 31 18:32:47.532394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.532370    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8d3cca1f-7dbd694677-8rssg"]
Oct 31 18:32:47.994902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.994865    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13097b93-539f-4187-86b4-10b649a9ef85" path="/var/lib/kubelet/pods/13097b93-539f-4187-86b4-10b649a9ef85/volumes"
Oct 31 18:32:47.995539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:47.995505    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a09f2c6-e58f-436b-a794-63b0da9037fb" path="/var/lib/kubelet/pods/9a09f2c6-e58f-436b-a794-63b0da9037fb/volumes"
Oct 31 18:32:48.124574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:48.124541    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:48.372237 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:48.372135    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:48.524541 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:48.524509    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:48.667424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:48.667318    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"]
Oct 31 18:32:48.719312 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:48.719276    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1f4b473d-j9d92"]
Oct 31 18:32:48.799920 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:48.799885    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:49.570676 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:49.570642    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:49.994657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:49.994619    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0570551-f242-44dd-a559-dd3d4daed62d" path="/var/lib/kubelet/pods/f0570551-f242-44dd-a559-dd3d4daed62d/volumes"
Oct 31 18:32:50.166538 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:50.166503    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:50.344447 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:50.344352    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:50.506820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:50.506736    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:53.468818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:53.468787    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:53.665634 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:53.665594    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:54.731567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:54.731529    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:54.904657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:54.904622    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:55.126044 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.126011    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"]
Oct 31 18:32:55.126545 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.126514    4484 cpu_manager.go:414] "containerMap: removing container" podUID="13097b93-539f-4187-86b4-10b649a9ef85" containerName="payload-processing"
Oct 31 18:32:55.126545 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.126534    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13097b93-539f-4187-86b4-10b649a9ef85" containerName="payload-processing"
Oct 31 18:32:55.126719 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.126598    4484 cpu_manager.go:414] "containerMap: removing container" podUID="9a09f2c6-e58f-436b-a794-63b0da9037fb" containerName="payload-pre-processing"
Oct 31 18:32:55.126719 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.126607    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a09f2c6-e58f-436b-a794-63b0da9037fb" containerName="payload-pre-processing"
Oct 31 18:32:55.126719 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.126712    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="13097b93-539f-4187-86b4-10b649a9ef85" containerName="payload-processing"
Oct 31 18:32:55.126862 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.126734    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a09f2c6-e58f-436b-a794-63b0da9037fb" containerName="payload-pre-processing"
Oct 31 18:32:55.134904 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.134884    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"
Oct 31 18:32:55.185022 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.184990    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"]
Oct 31 18:32:55.233005 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.232980    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-da6ea7\""
Oct 31 18:32:55.233005 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.233002    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-da6ea7-dockercfg-gmk6z\""
Oct 31 18:32:55.249291 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.249258    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ns6r\" (UniqueName: \"kubernetes.io/projected/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-kube-api-access-6ns6r\") pod \"payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6\" (UID: \"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"
Oct 31 18:32:55.249447 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.249434    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6\" (UID: \"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"
Oct 31 18:32:55.285646 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.285615    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"]
Oct 31 18:32:55.289767 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.289746    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"
Oct 31 18:32:55.342515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.342465    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"]
Oct 31 18:32:55.350964 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.350925    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6\" (UID: \"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"
Oct 31 18:32:55.351125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.350987    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dzqg\" (UniqueName: \"kubernetes.io/projected/486fd098-2d6b-4a72-a28a-b7203b42bb61-kube-api-access-6dzqg\") pod \"payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc\" (UID: \"486fd098-2d6b-4a72-a28a-b7203b42bb61\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"
Oct 31 18:32:55.351125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.351038    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ns6r\" (UniqueName: \"kubernetes.io/projected/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-kube-api-access-6ns6r\") pod \"payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6\" (UID: \"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"
Oct 31 18:32:55.351207 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.351127    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486fd098-2d6b-4a72-a28a-b7203b42bb61-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc\" (UID: \"486fd098-2d6b-4a72-a28a-b7203b42bb61\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"
Oct 31 18:32:55.351733 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.351713    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6\" (UID: \"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"
Oct 31 18:32:55.409706 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.409636    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ns6r\" (UniqueName: \"kubernetes.io/projected/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-kube-api-access-6ns6r\") pod \"payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6\" (UID: \"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"
Oct 31 18:32:55.447947 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.447913    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"
Oct 31 18:32:55.452270 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.452247    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dzqg\" (UniqueName: \"kubernetes.io/projected/486fd098-2d6b-4a72-a28a-b7203b42bb61-kube-api-access-6dzqg\") pod \"payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc\" (UID: \"486fd098-2d6b-4a72-a28a-b7203b42bb61\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"
Oct 31 18:32:55.452410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.452318    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486fd098-2d6b-4a72-a28a-b7203b42bb61-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc\" (UID: \"486fd098-2d6b-4a72-a28a-b7203b42bb61\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"
Oct 31 18:32:55.452977 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.452954    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486fd098-2d6b-4a72-a28a-b7203b42bb61-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc\" (UID: \"486fd098-2d6b-4a72-a28a-b7203b42bb61\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"
Oct 31 18:32:55.474118 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:32:55.474086    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:32:55.519119 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.519094    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dzqg\" (UniqueName: \"kubernetes.io/projected/486fd098-2d6b-4a72-a28a-b7203b42bb61-kube-api-access-6dzqg\") pod \"payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc\" (UID: \"486fd098-2d6b-4a72-a28a-b7203b42bb61\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"
Oct 31 18:32:55.575062 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.575027    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:55.604720 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.604685    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"
Oct 31 18:32:55.758649 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.758622    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:55.821985 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.821964    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"]
Oct 31 18:32:55.823980 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:32:55.823956    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff209d0_91a6_4b4e_9fd5_fe7258c12c96.slice/crio-4f504e987aa808bb9564404e960eeaa3c71faca71aaea3ec10405e6672d85c08 WatchSource:0}: Error finding container 4f504e987aa808bb9564404e960eeaa3c71faca71aaea3ec10405e6672d85c08: Status 404 returned error can't find the container with id 4f504e987aa808bb9564404e960eeaa3c71faca71aaea3ec10405e6672d85c08
Oct 31 18:32:55.930776 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.930683    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:55.980795 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:55.980766    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"]
Oct 31 18:32:55.982262 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:32:55.982237    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486fd098_2d6b_4a72_a28a_b7203b42bb61.slice/crio-30cb21c75a95c842a361992f1c5666549efab74cb1e7d86c4984d5b59cf990e2 WatchSource:0}: Error finding container 30cb21c75a95c842a361992f1c5666549efab74cb1e7d86c4984d5b59cf990e2: Status 404 returned error can't find the container with id 30cb21c75a95c842a361992f1c5666549efab74cb1e7d86c4984d5b59cf990e2
Oct 31 18:32:56.134157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:56.134094    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6" event={"ID":"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96","Type":"ContainerStarted","Data":"5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532"}
Oct 31 18:32:56.134347 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:56.134184    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"
Oct 31 18:32:56.134347 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:56.134210    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6" event={"ID":"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96","Type":"ContainerStarted","Data":"4f504e987aa808bb9564404e960eeaa3c71faca71aaea3ec10405e6672d85c08"}
Oct 31 18:32:56.135705 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:56.135675    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc" event={"ID":"486fd098-2d6b-4a72-a28a-b7203b42bb61","Type":"ContainerStarted","Data":"0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f"}
Oct 31 18:32:56.135837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:56.135727    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"
Oct 31 18:32:56.135837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:56.135747    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc" event={"ID":"486fd098-2d6b-4a72-a28a-b7203b42bb61","Type":"ContainerStarted","Data":"30cb21c75a95c842a361992f1c5666549efab74cb1e7d86c4984d5b59cf990e2"}
Oct 31 18:32:56.249614 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:56.249564    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6" podStartSLOduration=1.2495501469999999 podStartE2EDuration="1.249550147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:56.247922658 +0000 UTC m=+722.394372550" watchObservedRunningTime="2026-10-31 18:32:56.249550147 +0000 UTC m=+722.396000012"
Oct 31 18:32:56.366815 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:56.366748    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc" podStartSLOduration=1.366729559 podStartE2EDuration="1.366729559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:32:56.36337973 +0000 UTC m=+722.509829597" watchObservedRunningTime="2026-10-31 18:32:56.366729559 +0000 UTC m=+722.513179427"
Oct 31 18:32:57.934987 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:57.934951    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"]
Oct 31 18:32:57.935401 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:57.935145    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt" podUID="66c1aebe-1457-45d8-bf58-0193185f1f3e" containerName="maas-api" containerID="cri-o://22d458a9f0792914478ffd8442f8df350712784d942f97cd72fa23c350aca5a9" gracePeriod=30
Oct 31 18:32:58.056852 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.056812    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:58.090493 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.090441    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"
Oct 31 18:32:58.158763 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.157984    4484 generic.go:356] "Generic (PLEG): container finished" podID="66c1aebe-1457-45d8-bf58-0193185f1f3e" containerID="22d458a9f0792914478ffd8442f8df350712784d942f97cd72fa23c350aca5a9" exitCode=0
Oct 31 18:32:58.158763 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.158083    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt" event={"ID":"66c1aebe-1457-45d8-bf58-0193185f1f3e","Type":"ContainerDied","Data":"22d458a9f0792914478ffd8442f8df350712784d942f97cd72fa23c350aca5a9"}
Oct 31 18:32:58.242172 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.242140    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:32:58.395458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.395076    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-maas-api-tls\") pod \"66c1aebe-1457-45d8-bf58-0193185f1f3e\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") "
Oct 31 18:32:58.395458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.395164    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c1aebe-1457-45d8-bf58-0193185f1f3e-kube-api-access-hvzxs\" (UniqueName: \"kubernetes.io/projected/66c1aebe-1457-45d8-bf58-0193185f1f3e-kube-api-access-hvzxs\") pod \"66c1aebe-1457-45d8-bf58-0193185f1f3e\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") "
Oct 31 18:32:58.395458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.395297    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c1aebe-1457-45d8-bf58-0193185f1f3e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c1aebe-1457-45d8-bf58-0193185f1f3e-openshift-service-ca\") pod \"66c1aebe-1457-45d8-bf58-0193185f1f3e\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") "
Oct 31 18:32:58.395458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.395342    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-metrics-certs\") pod \"66c1aebe-1457-45d8-bf58-0193185f1f3e\" (UID: \"66c1aebe-1457-45d8-bf58-0193185f1f3e\") "
Oct 31 18:32:58.396373 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.396352    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c1aebe-1457-45d8-bf58-0193185f1f3e-openshift-service-ca" pod "66c1aebe-1457-45d8-bf58-0193185f1f3e" (UID: "66c1aebe-1457-45d8-bf58-0193185f1f3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:32:58.398166 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.398143    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-maas-api-tls" pod "66c1aebe-1457-45d8-bf58-0193185f1f3e" (UID: "66c1aebe-1457-45d8-bf58-0193185f1f3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:32:58.398322 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.398297    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-metrics-certs" pod "66c1aebe-1457-45d8-bf58-0193185f1f3e" (UID: "66c1aebe-1457-45d8-bf58-0193185f1f3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:32:58.402035 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.402008    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c1aebe-1457-45d8-bf58-0193185f1f3e-kube-api-access-hvzxs" pod "66c1aebe-1457-45d8-bf58-0193185f1f3e" (UID: "66c1aebe-1457-45d8-bf58-0193185f1f3e"). InnerVolumeSpecName "kube-api-access-hvzxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:32:58.496378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.496338    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:58.496378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.496382    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvzxs\" (UniqueName: \"kubernetes.io/projected/66c1aebe-1457-45d8-bf58-0193185f1f3e-kube-api-access-hvzxs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:58.496680 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.496401    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66c1aebe-1457-45d8-bf58-0193185f1f3e-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:58.496680 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:58.496415    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66c1aebe-1457-45d8-bf58-0193185f1f3e-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:32:59.190515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:59.188826    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt" event={"ID":"66c1aebe-1457-45d8-bf58-0193185f1f3e","Type":"ContainerDied","Data":"d0808b66bcc9bfa05581ba4aa4776ade629eaaa0291a5b6a452c5c5c52e97bf3"}
Oct 31 18:32:59.190515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:59.188887    4484 scope.go:122] "RemoveContainer" containerID="22d458a9f0792914478ffd8442f8df350712784d942f97cd72fa23c350aca5a9"
Oct 31 18:32:59.190515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:59.188910    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"
Oct 31 18:32:59.354826 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:59.354797    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"]
Oct 31 18:32:59.356693 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:59.356674    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8d3cca1f-85db465d45-qvwwt"]
Oct 31 18:32:59.883499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:59.882253    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:32:59.993154 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:32:59.993123    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c1aebe-1457-45d8-bf58-0193185f1f3e" path="/var/lib/kubelet/pods/66c1aebe-1457-45d8-bf58-0193185f1f3e/volumes"
Oct 31 18:33:00.050525 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:00.050494    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:00.201953 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:00.201825    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:00.347833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:00.347797    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"]
Oct 31 18:33:00.351033 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:00.351012    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8d3cca1f-jqjzc"]
Oct 31 18:33:00.924979 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:00.924948    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"]
Oct 31 18:33:00.925246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:00.925219    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy" containerID="cri-o://a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013" gracePeriod=30
Oct 31 18:33:00.957947 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:00.957920    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:01.798691 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:01.798659    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:01.997549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:01.997511    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d51e09a2-b0a3-40d9-889f-cab103a056ef" path="/var/lib/kubelet/pods/d51e09a2-b0a3-40d9-889f-cab103a056ef/volumes"
Oct 31 18:33:01.997972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:01.997948    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:02.165829 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.165797    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:02.258765 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.258722    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"]
Oct 31 18:33:02.259264 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.259244    4484 cpu_manager.go:414] "containerMap: removing container" podUID="66c1aebe-1457-45d8-bf58-0193185f1f3e" containerName="maas-api"
Oct 31 18:33:02.259354 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.259266    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c1aebe-1457-45d8-bf58-0193185f1f3e" containerName="maas-api"
Oct 31 18:33:02.259414 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.259401    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c1aebe-1457-45d8-bf58-0193185f1f3e" containerName="maas-api"
Oct 31 18:33:02.264072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.264045    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"
Oct 31 18:33:02.317091 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.317052    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-3e4a2c\""
Oct 31 18:33:02.341397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.341307    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"]
Oct 31 18:33:02.345212 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.345178    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:02.392253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.392222    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-3e4a2c-dockercfg-zbp8p\""
Oct 31 18:33:02.401606 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.401566    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"]
Oct 31 18:33:02.406186 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.406154    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"
Oct 31 18:33:02.439940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.439888    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82zsx\" (UniqueName: \"kubernetes.io/projected/50dad950-53be-49e3-883b-d803ac7858df-kube-api-access-82zsx\") pod \"payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws\" (UID: \"50dad950-53be-49e3-883b-d803ac7858df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"
Oct 31 18:33:02.440124 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.440017    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50dad950-53be-49e3-883b-d803ac7858df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws\" (UID: \"50dad950-53be-49e3-883b-d803ac7858df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"
Oct 31 18:33:02.458663 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.458622    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"]
Oct 31 18:33:02.541507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.541461    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q54f\" (UniqueName: \"kubernetes.io/projected/1142e654-6da9-46c3-a10e-c84bdd017f78-kube-api-access-9q54f\") pod \"payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz\" (UID: \"1142e654-6da9-46c3-a10e-c84bdd017f78\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"
Oct 31 18:33:02.541711 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.541567    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50dad950-53be-49e3-883b-d803ac7858df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws\" (UID: \"50dad950-53be-49e3-883b-d803ac7858df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"
Oct 31 18:33:02.541711 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.541692    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1142e654-6da9-46c3-a10e-c84bdd017f78-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz\" (UID: \"1142e654-6da9-46c3-a10e-c84bdd017f78\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"
Oct 31 18:33:02.541830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.541721    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82zsx\" (UniqueName: \"kubernetes.io/projected/50dad950-53be-49e3-883b-d803ac7858df-kube-api-access-82zsx\") pod \"payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws\" (UID: \"50dad950-53be-49e3-883b-d803ac7858df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"
Oct 31 18:33:02.542290 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.542265    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50dad950-53be-49e3-883b-d803ac7858df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws\" (UID: \"50dad950-53be-49e3-883b-d803ac7858df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"
Oct 31 18:33:02.599314 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.599233    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82zsx\" (UniqueName: \"kubernetes.io/projected/50dad950-53be-49e3-883b-d803ac7858df-kube-api-access-82zsx\") pod \"payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws\" (UID: \"50dad950-53be-49e3-883b-d803ac7858df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"
Oct 31 18:33:02.607506 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.607437    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:02.642502 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.642418    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1142e654-6da9-46c3-a10e-c84bdd017f78-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz\" (UID: \"1142e654-6da9-46c3-a10e-c84bdd017f78\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"
Oct 31 18:33:02.642711 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.642520    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q54f\" (UniqueName: \"kubernetes.io/projected/1142e654-6da9-46c3-a10e-c84bdd017f78-kube-api-access-9q54f\") pod \"payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz\" (UID: \"1142e654-6da9-46c3-a10e-c84bdd017f78\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"
Oct 31 18:33:02.643270 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.643248    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1142e654-6da9-46c3-a10e-c84bdd017f78-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz\" (UID: \"1142e654-6da9-46c3-a10e-c84bdd017f78\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"
Oct 31 18:33:02.700653 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.700618    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q54f\" (UniqueName: \"kubernetes.io/projected/1142e654-6da9-46c3-a10e-c84bdd017f78-kube-api-access-9q54f\") pod \"payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz\" (UID: \"1142e654-6da9-46c3-a10e-c84bdd017f78\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"
Oct 31 18:33:02.724327 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.724292    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"
Oct 31 18:33:02.883488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:02.883375    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"
Oct 31 18:33:03.101284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:03.101254    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"]
Oct 31 18:33:03.103125 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:33:03.103095    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1142e654_6da9_46c3_a10e_c84bdd017f78.slice/crio-9f7930501658e80466780b1e8bb170cbb7ecb237caa634233c3db2afa960dc9b WatchSource:0}: Error finding container 9f7930501658e80466780b1e8bb170cbb7ecb237caa634233c3db2afa960dc9b: Status 404 returned error can't find the container with id 9f7930501658e80466780b1e8bb170cbb7ecb237caa634233c3db2afa960dc9b
Oct 31 18:33:03.213785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:03.213752    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz" event={"ID":"1142e654-6da9-46c3-a10e-c84bdd017f78","Type":"ContainerStarted","Data":"a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe"}
Oct 31 18:33:03.213969 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:03.213818    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz" event={"ID":"1142e654-6da9-46c3-a10e-c84bdd017f78","Type":"ContainerStarted","Data":"9f7930501658e80466780b1e8bb170cbb7ecb237caa634233c3db2afa960dc9b"}
Oct 31 18:33:03.213969 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:03.213848    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"
Oct 31 18:33:03.265061 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:03.265029    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"]
Oct 31 18:33:03.265998 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:33:03.265964    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50dad950_53be_49e3_883b_d803ac7858df.slice/crio-024ea2734113e4ab2dfca833724b9363dc7a88d52c1bae7c7419d60fe3247324 WatchSource:0}: Error finding container 024ea2734113e4ab2dfca833724b9363dc7a88d52c1bae7c7419d60fe3247324: Status 404 returned error can't find the container with id 024ea2734113e4ab2dfca833724b9363dc7a88d52c1bae7c7419d60fe3247324
Oct 31 18:33:03.328394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:03.328333    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz" podStartSLOduration=1.328310964 podStartE2EDuration="1.328310964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:33:03.327343725 +0000 UTC m=+729.473793592" watchObservedRunningTime="2026-10-31 18:33:03.328310964 +0000 UTC m=+729.474760830"
Oct 31 18:33:03.877939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:03.877909    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:33:04.221183 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.221094    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws" event={"ID":"50dad950-53be-49e3-883b-d803ac7858df","Type":"ContainerStarted","Data":"bc2a0f564dec4353e855a72ef2421569b70401cda30ff554c8df100e5d991a64"}
Oct 31 18:33:04.221183 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.221160    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws" event={"ID":"50dad950-53be-49e3-883b-d803ac7858df","Type":"ContainerStarted","Data":"024ea2734113e4ab2dfca833724b9363dc7a88d52c1bae7c7419d60fe3247324"}
Oct 31 18:33:04.337069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.337016    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws" podStartSLOduration=2.337002434 podStartE2EDuration="2.337002434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:33:04.336607781 +0000 UTC m=+730.483057647" watchObservedRunningTime="2026-10-31 18:33:04.337002434 +0000 UTC m=+730.483452299"
Oct 31 18:33:04.520835 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.520795    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"]
Oct 31 18:33:04.524761 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.524738    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:04.581059 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.581022    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-da6ea7-serving-cert\""
Oct 31 18:33:04.581059 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.581055    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czkfc\""
Oct 31 18:33:04.581350 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.581211    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 18:33:04.583765 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.583743    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"]
Oct 31 18:33:04.663521 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.663465    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lbpv\" (UniqueName: \"kubernetes.io/projected/0575483e-549c-4b63-bc9f-72218b8d72d8-kube-api-access-5lbpv\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:04.663731 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.663676    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0575483e-549c-4b63-bc9f-72218b8d72d8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:04.663789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.663775    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-metrics-certs\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:04.663909 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.663882    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:04.747121 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.747089    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:04.765194 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.765158    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0575483e-549c-4b63-bc9f-72218b8d72d8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:04.765394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.765206    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-metrics-certs\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:04.765394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.765233    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:04.765394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.765281    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lbpv\" (UniqueName: \"kubernetes.io/projected/0575483e-549c-4b63-bc9f-72218b8d72d8-kube-api-access-5lbpv\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:04.765588 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:33:04.765398    4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-serving-cert: secret "maas-api-e2e-worker-w1-da6ea7-serving-cert" not found
Oct 31 18:33:04.765588 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:33:04.765468    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-maas-api-tls podName:0575483e-549c-4b63-bc9f-72218b8d72d8 nodeName:}" failed. No retries permitted until 2026-10-31 18:33:05.265448471 +0000 UTC m=+731.411898314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-maas-api-tls") pod "maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d" (UID: "0575483e-549c-4b63-bc9f-72218b8d72d8") : secret "maas-api-e2e-worker-w1-da6ea7-serving-cert" not found
Oct 31 18:33:04.765995 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.765973    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0575483e-549c-4b63-bc9f-72218b8d72d8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:04.768096 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.768071    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-metrics-certs\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:04.825261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.825179    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lbpv\" (UniqueName: \"kubernetes.io/projected/0575483e-549c-4b63-bc9f-72218b8d72d8-kube-api-access-5lbpv\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:04.950731 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:04.950686    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:05.108722 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:05.108621    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:05.227072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:05.227033    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"
Oct 31 18:33:05.270772 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:05.270733    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:05.273595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:05.273565    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:05.438353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:05.438261    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:05.808941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:05.808910    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"]
Oct 31 18:33:05.810637 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:33:05.810604    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0575483e_549c_4b63_bc9f_72218b8d72d8.slice/crio-a58bc63adc6b6040725350c5874b8e88fd0834a74414868b40ccabb16e362c6a WatchSource:0}: Error finding container a58bc63adc6b6040725350c5874b8e88fd0834a74414868b40ccabb16e362c6a: Status 404 returned error can't find the container with id a58bc63adc6b6040725350c5874b8e88fd0834a74414868b40ccabb16e362c6a
Oct 31 18:33:06.175326 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.175297    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:33:06.233450 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.233417    4484 generic.go:356] "Generic (PLEG): container finished" podID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerID="384153cc820925dec13341644dcad1a6b1aef5c5042eedddc2801d5968a62096" exitCode=0
Oct 31 18:33:06.233905 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.233499    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" event={"ID":"be7b417c-a740-4b1d-ab49-467a568ef17f","Type":"ContainerDied","Data":"384153cc820925dec13341644dcad1a6b1aef5c5042eedddc2801d5968a62096"}
Oct 31 18:33:06.233905 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.233899    4484 scope.go:122] "RemoveContainer" containerID="384153cc820925dec13341644dcad1a6b1aef5c5042eedddc2801d5968a62096"
Oct 31 18:33:06.235272 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.235243    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:33:06.235364 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.235271    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" event={"ID":"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe","Type":"ContainerDied","Data":"a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013"}
Oct 31 18:33:06.235364 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.235299    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"
Oct 31 18:33:06.235364 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.235311    4484 scope.go:122] "RemoveContainer" containerID="a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013"
Oct 31 18:33:06.235364 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.235321    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:33:06.235364 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.235355    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:33:06.235635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.235250    4484 generic.go:356] "Generic (PLEG): container finished" podID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerID="a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013" exitCode=0
Oct 31 18:33:06.235635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.235436    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w" event={"ID":"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe","Type":"ContainerDied","Data":"23f970af9016109a98b75d633ba76d63e9e3ac267d150298c53cf1a4a3ff02c3"}
Oct 31 18:33:06.236782 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.236762    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d" event={"ID":"0575483e-549c-4b63-bc9f-72218b8d72d8","Type":"ContainerStarted","Data":"a58bc63adc6b6040725350c5874b8e88fd0834a74414868b40ccabb16e362c6a"}
Oct 31 18:33:06.245653 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.245627    4484 scope.go:122] "RemoveContainer" containerID="07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3"
Oct 31 18:33:06.254920 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.254895    4484 scope.go:122] "RemoveContainer" containerID="a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013"
Oct 31 18:33:06.255360 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:33:06.255340    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013\": container with ID starting with a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013 not found: ID does not exist" containerID="a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013"
Oct 31 18:33:06.255457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.255372    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013"} err="failed to get container status \"a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013\": rpc error: code = NotFound desc = could not find container \"a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013\": container with ID starting with a8964db0d9e376c4712cfb6aec4faff2fb3eff8d98afde9b92728720ac879013 not found: ID does not exist"
Oct 31 18:33:06.255457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.255401    4484 scope.go:122] "RemoveContainer" containerID="07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3"
Oct 31 18:33:06.255763 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:33:06.255744    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3\": container with ID starting with 07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3 not found: ID does not exist" containerID="07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3"
Oct 31 18:33:06.255809 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.255768    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3"} err="failed to get container status \"07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3\": rpc error: code = NotFound desc = could not find container \"07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3\": container with ID starting with 07f610e3810a486b8f4d63388d45d85b4b639c6424985eba6438c318b3392aa3 not found: ID does not exist"
Oct 31 18:33:06.281763 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.281738    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-data\") pod \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") "
Oct 31 18:33:06.281916 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.281774    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-envoy\") pod \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") "
Oct 31 18:33:06.281916 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.281807    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-kube-api-access-c9k8k\" (UniqueName: \"kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-kube-api-access-c9k8k\") pod \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") "
Oct 31 18:33:06.281916 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.281842    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istiod-ca-cert\") pod \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") "
Oct 31 18:33:06.281916 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.281879    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-socket\") pod \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") "
Oct 31 18:33:06.281916 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.281901    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-credential-socket\") pod \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") "
Oct 31 18:33:06.282169 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.281942    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-podinfo\") pod \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") "
Oct 31 18:33:06.282169 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.281977    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-certs\") pod \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") "
Oct 31 18:33:06.282169 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.282054    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-data" pod "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" (UID: "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:33:06.282169 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.282108    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-token\" (UniqueName: \"kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-token\") pod \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\" (UID: \"beec46bd-b811-4263-b2f4-ba6f4f6d8ffe\") "
Oct 31 18:33:06.282375 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.282281    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-credential-socket" pod "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" (UID: "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:33:06.282375 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.282323    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istiod-ca-cert" pod "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" (UID: "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:33:06.282548 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.282528    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:33:06.282548 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.282536    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-socket" pod "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" (UID: "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:33:06.282700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.282557    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:33:06.282700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.282579    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:33:06.282803 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.282767    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-certs" pod "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" (UID: "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:33:06.284066 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.284035    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-envoy" pod "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" (UID: "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:33:06.284417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.284365    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-podinfo" pod "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" (UID: "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:33:06.284555 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.284517    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-token" pod "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" (UID: "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:33:06.284607 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.284569    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-kube-api-access-c9k8k" pod "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" (UID: "beec46bd-b811-4263-b2f4-ba6f4f6d8ffe"). InnerVolumeSpecName "kube-api-access-c9k8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:33:06.384142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.384041    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:33:06.384142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.384083    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:33:06.384142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.384098    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9k8k\" (UniqueName: \"kubernetes.io/projected/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-kube-api-access-c9k8k\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:33:06.384142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.384114    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:33:06.384142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.384129    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:33:06.384142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.384143    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:33:06.657026 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.656997    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"]
Oct 31 18:33:06.663165 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:06.663131    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8d3cca1f-openshift-default-7b54bbccc5-86w5w"]
Oct 31 18:33:07.143895 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.143869    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"
Oct 31 18:33:07.144068 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.143935    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"
Oct 31 18:33:07.187518 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.187360    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:07.245392 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.245356    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" event={"ID":"be7b417c-a740-4b1d-ab49-467a568ef17f","Type":"ContainerStarted","Data":"6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d"}
Oct 31 18:33:07.248722 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.248692    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d" event={"ID":"0575483e-549c-4b63-bc9f-72218b8d72d8","Type":"ContainerStarted","Data":"2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f"}
Oct 31 18:33:07.248873 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.248767    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:07.354407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.354380    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:07.673503 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.662044    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d" podStartSLOduration=2.8187395410000002 podStartE2EDuration="3.662024147s" totalImagesPullingTime="843.284606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:33:07.659419201 +0000 UTC m=+733.805869065" watchObservedRunningTime="2026-10-31 18:33:07.662024147 +0000 UTC m=+733.808474013"
Oct 31 18:33:07.693790 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.693755    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:33:07.695120 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.695096    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 18:33:07.695267 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.695150    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 18:33:07.740896 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.740860    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:07.993619 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:07.993587    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" path="/var/lib/kubelet/pods/beec46bd-b811-4263-b2f4-ba6f4f6d8ffe/volumes"
Oct 31 18:33:08.693980 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:08.693941    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 18:33:08.694368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:08.693998    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 18:33:09.084624 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:09.084525    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:09.682149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:09.682110    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:09.694024 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:09.693991    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 18:33:09.694548 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:09.694049    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 18:33:09.848716 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:09.848684    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:10.307501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:10.306689    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:10.693438 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:10.693354    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 18:33:10.693438 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:10.693414    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 18:33:11.693575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:11.693538    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 18:33:11.693996 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:11.693602    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 18:33:11.947351 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:11.947259    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:33:12.697762 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:12.697734    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:33:12.698593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:12.698573    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 18:33:13.065213 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:13.065179    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:13.213418 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:13.213350    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:33:13.260520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:13.260493    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 18:33:13.582901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:13.582857    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:13.752670 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:13.752593    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:13.974039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:13.973955    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"]
Oct 31 18:33:13.974556 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:13.974538    4484 cpu_manager.go:414] "containerMap: removing container" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy"
Oct 31 18:33:13.974556 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:13.974556    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy"
Oct 31 18:33:13.974665 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:13.974646    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy"
Oct 31 18:33:13.977823 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:13.977806    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"
Oct 31 18:33:14.033092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.033061    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a8587d\""
Oct 31 18:33:14.033378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.033315    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a8587d-dockercfg-kq6rv\""
Oct 31 18:33:14.036458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.036426    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"]
Oct 31 18:33:14.067033 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.066992    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8sxs\" (UniqueName: \"kubernetes.io/projected/65891533-3ba8-44da-b413-d6936580f382-kube-api-access-l8sxs\") pod \"payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg\" (UID: \"65891533-3ba8-44da-b413-d6936580f382\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"
Oct 31 18:33:14.067253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.067191    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65891533-3ba8-44da-b413-d6936580f382-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg\" (UID: \"65891533-3ba8-44da-b413-d6936580f382\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"
Oct 31 18:33:14.124497 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.124440    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"]
Oct 31 18:33:14.124943 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.124930    4484 cpu_manager.go:414] "containerMap: removing container" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy"
Oct 31 18:33:14.124986 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.124944    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy"
Oct 31 18:33:14.125036 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.125028    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="beec46bd-b811-4263-b2f4-ba6f4f6d8ffe" containerName="istio-proxy"
Oct 31 18:33:14.128119 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.128098    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"
Oct 31 18:33:14.167917 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.167876    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65891533-3ba8-44da-b413-d6936580f382-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg\" (UID: \"65891533-3ba8-44da-b413-d6936580f382\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"
Oct 31 18:33:14.168084 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.167951    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b43632-5dbb-4491-b1b8-48c1383f7266-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk\" (UID: \"73b43632-5dbb-4491-b1b8-48c1383f7266\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"
Oct 31 18:33:14.168084 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.167987    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ccd7\" (UniqueName: \"kubernetes.io/projected/73b43632-5dbb-4491-b1b8-48c1383f7266-kube-api-access-6ccd7\") pod \"payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk\" (UID: \"73b43632-5dbb-4491-b1b8-48c1383f7266\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"
Oct 31 18:33:14.168187 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.168089    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8sxs\" (UniqueName: \"kubernetes.io/projected/65891533-3ba8-44da-b413-d6936580f382-kube-api-access-l8sxs\") pod \"payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg\" (UID: \"65891533-3ba8-44da-b413-d6936580f382\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"
Oct 31 18:33:14.168608 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.168592    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65891533-3ba8-44da-b413-d6936580f382-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg\" (UID: \"65891533-3ba8-44da-b413-d6936580f382\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"
Oct 31 18:33:14.184029 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.183998    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"]
Oct 31 18:33:14.233451 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.233359    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8sxs\" (UniqueName: \"kubernetes.io/projected/65891533-3ba8-44da-b413-d6936580f382-kube-api-access-l8sxs\") pod \"payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg\" (UID: \"65891533-3ba8-44da-b413-d6936580f382\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"
Oct 31 18:33:14.233451 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.233409    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"
Oct 31 18:33:14.268995 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.268950    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b43632-5dbb-4491-b1b8-48c1383f7266-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk\" (UID: \"73b43632-5dbb-4491-b1b8-48c1383f7266\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"
Oct 31 18:33:14.269194 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.269026    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ccd7\" (UniqueName: \"kubernetes.io/projected/73b43632-5dbb-4491-b1b8-48c1383f7266-kube-api-access-6ccd7\") pod \"payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk\" (UID: \"73b43632-5dbb-4491-b1b8-48c1383f7266\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"
Oct 31 18:33:14.269699 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.269675    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b43632-5dbb-4491-b1b8-48c1383f7266-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk\" (UID: \"73b43632-5dbb-4491-b1b8-48c1383f7266\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"
Oct 31 18:33:14.291754 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.291724    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"
Oct 31 18:33:14.326283 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.326251    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ccd7\" (UniqueName: \"kubernetes.io/projected/73b43632-5dbb-4491-b1b8-48c1383f7266-kube-api-access-6ccd7\") pod \"payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk\" (UID: \"73b43632-5dbb-4491-b1b8-48c1383f7266\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"
Oct 31 18:33:14.441754 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.441717    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"
Oct 31 18:33:14.664497 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.664455    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"]
Oct 31 18:33:14.666059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:33:14.666035    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65891533_3ba8_44da_b413_d6936580f382.slice/crio-009325fe699fb633e58bdefe74556b610fed139c8373d6ae3cc0b0071473059b WatchSource:0}: Error finding container 009325fe699fb633e58bdefe74556b610fed139c8373d6ae3cc0b0071473059b: Status 404 returned error can't find the container with id 009325fe699fb633e58bdefe74556b610fed139c8373d6ae3cc0b0071473059b
Oct 31 18:33:14.813834 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:14.813809    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"]
Oct 31 18:33:14.815555 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:33:14.815527    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b43632_5dbb_4491_b1b8_48c1383f7266.slice/crio-b25d7d7772d9f8dc146b88d3200b9c8187f866d8342e387ed7a0b9cf36842cb0 WatchSource:0}: Error finding container b25d7d7772d9f8dc146b88d3200b9c8187f866d8342e387ed7a0b9cf36842cb0: Status 404 returned error can't find the container with id b25d7d7772d9f8dc146b88d3200b9c8187f866d8342e387ed7a0b9cf36842cb0
Oct 31 18:33:15.315440 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:15.315404    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk" event={"ID":"73b43632-5dbb-4491-b1b8-48c1383f7266","Type":"ContainerStarted","Data":"9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a"}
Oct 31 18:33:15.315621 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:15.315461    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk" event={"ID":"73b43632-5dbb-4491-b1b8-48c1383f7266","Type":"ContainerStarted","Data":"b25d7d7772d9f8dc146b88d3200b9c8187f866d8342e387ed7a0b9cf36842cb0"}
Oct 31 18:33:15.315621 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:15.315507    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"
Oct 31 18:33:15.316977 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:15.316952    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg" event={"ID":"65891533-3ba8-44da-b413-d6936580f382","Type":"ContainerStarted","Data":"8aaf1882962d417e01bc9eb10b9956b6ca5f99d9c5156515f80044d97c7f0203"}
Oct 31 18:33:15.317125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:15.316998    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg" event={"ID":"65891533-3ba8-44da-b413-d6936580f382","Type":"ContainerStarted","Data":"009325fe699fb633e58bdefe74556b610fed139c8373d6ae3cc0b0071473059b"}
Oct 31 18:33:15.317125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:15.317028    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"
Oct 31 18:33:15.421257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:15.421225    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:15.432321 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:15.432271    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk" podStartSLOduration=1.432258605 podStartE2EDuration="1.432258605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:33:15.430953823 +0000 UTC m=+741.577403687" watchObservedRunningTime="2026-10-31 18:33:15.432258605 +0000 UTC m=+741.578708469"
Oct 31 18:33:15.548435 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:15.548381    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg" podStartSLOduration=2.548367163 podStartE2EDuration="2.548367163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:33:15.54784484 +0000 UTC m=+741.694294705" watchObservedRunningTime="2026-10-31 18:33:15.548367163 +0000 UTC m=+741.694817028"
Oct 31 18:33:15.628268 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:15.628183    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:16.240714 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:16.240680    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"
Oct 31 18:33:20.367534 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:20.367495    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:22.496835 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:22.496803    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:33:25.074592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:25.074556    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:25.468831 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:33:25.468747    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:33:26.326054 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:26.326024    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"
Oct 31 18:33:26.326509 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:26.326460    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"
Oct 31 18:33:26.945092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:26.945057    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:27.531817 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:27.531772    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:27.697938 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:27.697904    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:30.249878 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:30.249849    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:34.711181 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.711144    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"]
Oct 31 18:33:34.716261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.716244    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:34.819683 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.819639    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-a8587d-serving-cert\""
Oct 31 18:33:34.821860 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.821833    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"]
Oct 31 18:33:34.880735 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.880697    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:34.881100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.881078    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24073104-4724-49d2-8056-64b9ffc3870b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:34.881299 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.881285    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlcn\" (UniqueName: \"kubernetes.io/projected/24073104-4724-49d2-8056-64b9ffc3870b-kube-api-access-2mlcn\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:34.881467 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.881453    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-metrics-certs\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:34.922296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.922265    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:34.982421 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.982329    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-metrics-certs\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:34.982421 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.982382    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:34.982421 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.982424    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24073104-4724-49d2-8056-64b9ffc3870b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:34.982739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.982503    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlcn\" (UniqueName: \"kubernetes.io/projected/24073104-4724-49d2-8056-64b9ffc3870b-kube-api-access-2mlcn\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:34.982739 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:33:34.982584    4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-serving-cert: secret "maas-api-e2e-worker-w0-a8587d-serving-cert" not found
Oct 31 18:33:34.982739 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:33:34.982659    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-maas-api-tls podName:24073104-4724-49d2-8056-64b9ffc3870b nodeName:}" failed. No retries permitted until 2026-10-31 18:33:35.482638801 +0000 UTC m=+761.629088644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-maas-api-tls") pod "maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq" (UID: "24073104-4724-49d2-8056-64b9ffc3870b") : secret "maas-api-e2e-worker-w0-a8587d-serving-cert" not found
Oct 31 18:33:34.983203 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.983185    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24073104-4724-49d2-8056-64b9ffc3870b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:34.985028 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:34.985005    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-metrics-certs\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:35.040298 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:35.040267    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlcn\" (UniqueName: \"kubernetes.io/projected/24073104-4724-49d2-8056-64b9ffc3870b-kube-api-access-2mlcn\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:35.079401 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:35.079367    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:35.288123 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:35.288085    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:35.488082 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:35.488042    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:35.490635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:35.490612    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:35.629654 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:35.629563    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:36.002755 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:36.002722    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"]
Oct 31 18:33:36.004374 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:33:36.004347    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24073104_4724_49d2_8056_64b9ffc3870b.slice/crio-6aad574058cc8f77789e1811797f14b012205f53eea64defe28b3e4cafcc5d93 WatchSource:0}: Error finding container 6aad574058cc8f77789e1811797f14b012205f53eea64defe28b3e4cafcc5d93: Status 404 returned error can't find the container with id 6aad574058cc8f77789e1811797f14b012205f53eea64defe28b3e4cafcc5d93
Oct 31 18:33:36.436573 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:36.436460    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq" event={"ID":"24073104-4724-49d2-8056-64b9ffc3870b","Type":"ContainerStarted","Data":"6aad574058cc8f77789e1811797f14b012205f53eea64defe28b3e4cafcc5d93"}
Oct 31 18:33:37.388845 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:37.388809    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:37.443531 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:37.443494    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq" event={"ID":"24073104-4724-49d2-8056-64b9ffc3870b","Type":"ContainerStarted","Data":"84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac"}
Oct 31 18:33:37.443703 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:37.443660    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:37.571064 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:37.571007    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq" podStartSLOduration=2.74599941 podStartE2EDuration="3.570993157s" totalImagesPullingTime="824.993747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:33:37.568979339 +0000 UTC m=+763.715429218" watchObservedRunningTime="2026-10-31 18:33:37.570993157 +0000 UTC m=+763.717443022"
Oct 31 18:33:39.719799 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:39.719764    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:40.273743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:40.273704    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:40.492005 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:40.491966    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:43.455398 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:43.455366    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 18:33:44.061486 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:44.061443    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:33:44.454633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:44.454551    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:45.068523 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:45.068488    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:49.760060 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:49.760021    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:54.668237 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:54.668200    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:55.037149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:55.037111    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:55.467944 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:33:55.467860    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:33:55.656381 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:55.656337    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:55.825433 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:55.825397    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:56.784274 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:56.784207    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:33:59.465042 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:33:59.465013    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:02.259237 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:02.259201    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:34:04.179314 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:04.179279    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:05.792976 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:05.792945    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:08.962244 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:08.962207    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:12.267446 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:12.267412    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:12.841198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:12.841162    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:13.001968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:13.001936    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:14.092189 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:14.092146    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:15.839802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:15.839771    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"]
Oct 31 18:34:15.843645 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:15.843628    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:15.947441 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:15.947400    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3e4a2c\"/\"kube-root-ca.crt\""
Oct 31 18:34:15.947441 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:15.947416    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3e4a2c\"/\"openshift-service-ca.crt\""
Oct 31 18:34:15.947658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:15.947464    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3e4a2c\"/\"default-dockercfg-x974w\""
Oct 31 18:34:15.947658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:15.947592    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3e4a2c\"/\"simulator-w3-3e4a2c-kserve-self-signed-certs\""
Oct 31 18:34:15.949695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:15.949676    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"]
Oct 31 18:34:15.953950 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:15.953915    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:16.006437 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.006401    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6b07cc-6154-4582-8e24-fc085949657a-tls-certs\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.006437 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.006435    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-home\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.006678 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.006456    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwnjf\" (UniqueName: \"kubernetes.io/projected/3b6b07cc-6154-4582-8e24-fc085949657a-kube-api-access-dwnjf\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.006678 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.006594    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-tmp-dir\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.006748 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.006693    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-dshm\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.006748 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.006724    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-model-cache\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.108273 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.108175    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-tmp-dir\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.108273 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.108240    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-dshm\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.108273 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.108274    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-model-cache\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.108617 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.108358    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6b07cc-6154-4582-8e24-fc085949657a-tls-certs\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.108617 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.108385    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-home\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.108617 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.108414    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwnjf\" (UniqueName: \"kubernetes.io/projected/3b6b07cc-6154-4582-8e24-fc085949657a-kube-api-access-dwnjf\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.108764 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.108715    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-tmp-dir\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.108817 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.108791    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-model-cache\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.108950 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.108928    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-home\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.111287 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.111260    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-dshm\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.111544 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.111527    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6b07cc-6154-4582-8e24-fc085949657a-tls-certs\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.165760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.165731    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwnjf\" (UniqueName: \"kubernetes.io/projected/3b6b07cc-6154-4582-8e24-fc085949657a-kube-api-access-dwnjf\") pod \"simulator-w3-3e4a2c-kserve-86d47f776b-qblsv\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") " pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.454970 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.454883    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:16.827817 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:16.827789    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"]
Oct 31 18:34:16.829643 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:34:16.829617    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b6b07cc_6154_4582_8e24_fc085949657a.slice/crio-b4a8852bef2639c6b1b814673575b3e7c864b3c2ee5dbed393aac518be20d583 WatchSource:0}: Error finding container b4a8852bef2639c6b1b814673575b3e7c864b3c2ee5dbed393aac518be20d583: Status 404 returned error can't find the container with id b4a8852bef2639c6b1b814673575b3e7c864b3c2ee5dbed393aac518be20d583
Oct 31 18:34:17.659788 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:17.659750    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv" event={"ID":"3b6b07cc-6154-4582-8e24-fc085949657a","Type":"ContainerStarted","Data":"d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926"}
Oct 31 18:34:17.660188 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:17.659800    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv" event={"ID":"3b6b07cc-6154-4582-8e24-fc085949657a","Type":"ContainerStarted","Data":"b4a8852bef2639c6b1b814673575b3e7c864b3c2ee5dbed393aac518be20d583"}
Oct 31 18:34:17.776997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:17.776944    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv" podStartSLOduration=2.776929537 podStartE2EDuration="2.776929537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:34:17.774610418 +0000 UTC m=+803.921060282" watchObservedRunningTime="2026-10-31 18:34:17.776929537 +0000 UTC m=+803.923379429"
Oct 31 18:34:18.455489 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:18.455445    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:18.758084 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:18.758051    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:20.939048 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:20.939005    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:34:23.462148 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:23.462110    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:25.467985 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:34:25.467944    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:34:26.455356 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:26.455306    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:26.467910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:26.467885    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:26.718169 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:26.718083    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:34:28.141151 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:28.141111    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:30.042440 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:30.042399    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:32.862177 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:32.862128    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:37.562033 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:37.561994    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:42.258867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.258826    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:42.802774 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.802741    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"]
Oct 31 18:34:42.806462 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.806435    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.862150 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.862115    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"]
Oct 31 18:34:42.875218 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.875183    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2ff\" (UniqueName: \"kubernetes.io/projected/a91dc442-830a-4436-bab0-fedc94f75038-kube-api-access-4r2ff\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.875396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.875261    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-home\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.875437 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.875376    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91dc442-830a-4436-bab0-fedc94f75038-tls-certs\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.875437 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.875420    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-model-cache\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.875565 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.875453    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-dshm\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.875667 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.875649    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-tmp-dir\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.881882 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.881861    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"openshift-service-ca.crt\""
Oct 31 18:34:42.910127 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.910093    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"kube-root-ca.crt\""
Oct 31 18:34:42.910331 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.910236    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"simulator-w0-a8587d-kserve-self-signed-certs\""
Oct 31 18:34:42.910331 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.910267    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a8587d\"/\"default-dockercfg-dmdhf\""
Oct 31 18:34:42.915348 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.915318    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:42.976629 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.976600    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91dc442-830a-4436-bab0-fedc94f75038-tls-certs\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.976629 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.976632    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-model-cache\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.976866 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.976651    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-dshm\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.976866 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.976700    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-tmp-dir\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.976866 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.976738    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2ff\" (UniqueName: \"kubernetes.io/projected/a91dc442-830a-4436-bab0-fedc94f75038-kube-api-access-4r2ff\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.976866 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.976792    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-home\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.977078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.977061    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-model-cache\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.977165 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.977140    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-tmp-dir\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.977209 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.977162    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-home\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.979058 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.979030    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-dshm\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:42.979330 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:42.979311    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91dc442-830a-4436-bab0-fedc94f75038-tls-certs\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:43.034263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:43.034228    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2ff\" (UniqueName: \"kubernetes.io/projected/a91dc442-830a-4436-bab0-fedc94f75038-kube-api-access-4r2ff\") pod \"simulator-w0-a8587d-kserve-f887488bb-d62z6\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") " pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:43.118837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:43.118746    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:43.503168 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:43.503143    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"]
Oct 31 18:34:43.504660 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:34:43.504628    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda91dc442_830a_4436_bab0_fedc94f75038.slice/crio-393b148f1f838dff58d9a6fb506dfad70c68459cc712cc1cf0be0c7aa0f6fd01 WatchSource:0}: Error finding container 393b148f1f838dff58d9a6fb506dfad70c68459cc712cc1cf0be0c7aa0f6fd01: Status 404 returned error can't find the container with id 393b148f1f838dff58d9a6fb506dfad70c68459cc712cc1cf0be0c7aa0f6fd01
Oct 31 18:34:43.798505 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:43.798390    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6" event={"ID":"a91dc442-830a-4436-bab0-fedc94f75038","Type":"ContainerStarted","Data":"04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce"}
Oct 31 18:34:43.798505 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:43.798430    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6" event={"ID":"a91dc442-830a-4436-bab0-fedc94f75038","Type":"ContainerStarted","Data":"393b148f1f838dff58d9a6fb506dfad70c68459cc712cc1cf0be0c7aa0f6fd01"}
Oct 31 18:34:44.048832 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:44.048731    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6" podStartSLOduration=2.048716203 podStartE2EDuration="2.048716203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:34:44.047248407 +0000 UTC m=+830.193698270" watchObservedRunningTime="2026-10-31 18:34:44.048716203 +0000 UTC m=+830.195166066"
Oct 31 18:34:44.119154 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:44.119116    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:44.697509 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:44.697460    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:47.149182 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:47.149151    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:51.924893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:51.924859    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:34:53.118988 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:53.118960    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:53.131363 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:53.131337    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:53.864515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:53.864486    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:34:55.468709 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:34:55.468661    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:34:56.549781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:34:56.549747    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:01.255598 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:01.255561    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:01.310277 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:01.310246    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:35:05.956697 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:05.956613    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:06.694655 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:06.694614    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:35:08.653399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:08.653359    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:09.257075 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:09.257041    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:10.656665 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:10.656617    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:15.374724 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:15.374686    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:18.172258 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:18.172225    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:35:20.091745 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:20.091701    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:24.747791 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:24.747746    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:25.469014 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:35:25.468975    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:35:29.476714 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:29.476673    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:32.999594 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:32.999562    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"]
Oct 31 18:35:33.003871 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.003851    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.058676 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.058641    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"]
Oct 31 18:35:33.092074 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.092032    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-model-cache\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.092251 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.092189    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dm8k\" (UniqueName: \"kubernetes.io/projected/020ad937-bef8-41ae-9394-eab2f30eefba-kube-api-access-7dm8k\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.092391 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.092375    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-home\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.092538 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.092516    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-dshm\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.092625 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.092588    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-tmp-dir\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.092680 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.092635    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020ad937-bef8-41ae-9394-eab2f30eefba-tls-certs\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.106194 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.106170    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3e4a2c\"/\"premium-w3-3e4a2c-kserve-self-signed-certs\""
Oct 31 18:35:33.110020 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.109992    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:33.193307 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.193272    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-home\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.193513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.193317    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-dshm\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.193513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.193367    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-tmp-dir\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.193513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.193385    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020ad937-bef8-41ae-9394-eab2f30eefba-tls-certs\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.193513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.193488    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-model-cache\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.193693 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.193525    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dm8k\" (UniqueName: \"kubernetes.io/projected/020ad937-bef8-41ae-9394-eab2f30eefba-kube-api-access-7dm8k\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.193813 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.193777    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-home\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.193813 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.193812    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-tmp-dir\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.193988 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.193958    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-model-cache\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.195830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.195811    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-dshm\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.196144 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.196123    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020ad937-bef8-41ae-9394-eab2f30eefba-tls-certs\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.251438 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.251374    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dm8k\" (UniqueName: \"kubernetes.io/projected/020ad937-bef8-41ae-9394-eab2f30eefba-kube-api-access-7dm8k\") pod \"premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") " pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.315934 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.315905    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:33.696715 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:33.696387    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"]
Oct 31 18:35:33.697900 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:35:33.697858    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod020ad937_bef8_41ae_9394_eab2f30eefba.slice/crio-88ca0d352d1b1cb2af1a07fff749cb85580df2332084dfe3fb6f8bfcf916c898 WatchSource:0}: Error finding container 88ca0d352d1b1cb2af1a07fff749cb85580df2332084dfe3fb6f8bfcf916c898: Status 404 returned error can't find the container with id 88ca0d352d1b1cb2af1a07fff749cb85580df2332084dfe3fb6f8bfcf916c898
Oct 31 18:35:34.062533 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:34.062492    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr" event={"ID":"020ad937-bef8-41ae-9394-eab2f30eefba","Type":"ContainerStarted","Data":"d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae"}
Oct 31 18:35:34.062533 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:34.062541    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr" event={"ID":"020ad937-bef8-41ae-9394-eab2f30eefba","Type":"ContainerStarted","Data":"88ca0d352d1b1cb2af1a07fff749cb85580df2332084dfe3fb6f8bfcf916c898"}
Oct 31 18:35:34.152242 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:34.152210    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:34.181624 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:34.181570    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr" podStartSLOduration=2.181556124 podStartE2EDuration="2.181556124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:35:34.179198472 +0000 UTC m=+880.325648339" watchObservedRunningTime="2026-10-31 18:35:34.181556124 +0000 UTC m=+880.328005988"
Oct 31 18:35:34.316513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:34.316423    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:38.883205 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:38.883169    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:39.709741 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:39.709703    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:35:41.136622 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:41.136578    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:43.316515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:43.316453    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:43.328743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:43.328717    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:43.572904 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:43.572805    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:44.128067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:44.128040    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:35:45.025433 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:45.025399    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:45.354840 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:45.354753    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:48.330725 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:48.330691    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:49.706882 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:49.706849    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:49.753545 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:49.753513    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"]
Oct 31 18:35:49.753790 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:49.753760    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" containerName="istio-proxy" containerID="cri-o://1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67" gracePeriod=30
Oct 31 18:35:53.252852 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:53.252817    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:35:54.081940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:54.081912    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:35:54.087621 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:54.087599    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:35:55.013355 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.013325    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:35:55.124078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.123977    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-socket\") pod \"675a6c4e-5415-4741-b381-257fd4c918c1\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") "
Oct 31 18:35:55.124078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124052    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-kube-api-access-bdpzn\" (UniqueName: \"kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-kube-api-access-bdpzn\") pod \"675a6c4e-5415-4741-b381-257fd4c918c1\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") "
Oct 31 18:35:55.124340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124086    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-envoy\") pod \"675a6c4e-5415-4741-b381-257fd4c918c1\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") "
Oct 31 18:35:55.124340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124141    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/675a6c4e-5415-4741-b381-257fd4c918c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675a6c4e-5415-4741-b381-257fd4c918c1-istio-podinfo\") pod \"675a6c4e-5415-4741-b381-257fd4c918c1\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") "
Oct 31 18:35:55.124340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124181    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-credential-socket\") pod \"675a6c4e-5415-4741-b381-257fd4c918c1\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") "
Oct 31 18:35:55.124340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124211    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/675a6c4e-5415-4741-b381-257fd4c918c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675a6c4e-5415-4741-b381-257fd4c918c1-istiod-ca-cert\") pod \"675a6c4e-5415-4741-b381-257fd4c918c1\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") "
Oct 31 18:35:55.124340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124299    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-socket" pod "675a6c4e-5415-4741-b381-257fd4c918c1" (UID: "675a6c4e-5415-4741-b381-257fd4c918c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:35:55.124340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124334    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-data\") pod \"675a6c4e-5415-4741-b381-257fd4c918c1\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") "
Oct 31 18:35:55.124728 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124364    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-istio-token\" (UniqueName: \"kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-istio-token\") pod \"675a6c4e-5415-4741-b381-257fd4c918c1\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") "
Oct 31 18:35:55.124728 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124397    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-certs\") pod \"675a6c4e-5415-4741-b381-257fd4c918c1\" (UID: \"675a6c4e-5415-4741-b381-257fd4c918c1\") "
Oct 31 18:35:55.124728 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124593    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-credential-socket" pod "675a6c4e-5415-4741-b381-257fd4c918c1" (UID: "675a6c4e-5415-4741-b381-257fd4c918c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:35:55.124865 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124725    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675a6c4e-5415-4741-b381-257fd4c918c1-istiod-ca-cert" pod "675a6c4e-5415-4741-b381-257fd4c918c1" (UID: "675a6c4e-5415-4741-b381-257fd4c918c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:35:55.124865 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124824    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-certs" pod "675a6c4e-5415-4741-b381-257fd4c918c1" (UID: "675a6c4e-5415-4741-b381-257fd4c918c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:35:55.125058 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124927    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.125058 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124947    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.125058 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124962    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.125058 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.124971    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/675a6c4e-5415-4741-b381-257fd4c918c1-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.125338 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.125163    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-data" pod "675a6c4e-5415-4741-b381-257fd4c918c1" (UID: "675a6c4e-5415-4741-b381-257fd4c918c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:35:55.126694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.126673    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-kube-api-access-bdpzn" pod "675a6c4e-5415-4741-b381-257fd4c918c1" (UID: "675a6c4e-5415-4741-b381-257fd4c918c1"). InnerVolumeSpecName "kube-api-access-bdpzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:35:55.126854 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.126830    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-istio-token" pod "675a6c4e-5415-4741-b381-257fd4c918c1" (UID: "675a6c4e-5415-4741-b381-257fd4c918c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:35:55.126902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.126879    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/675a6c4e-5415-4741-b381-257fd4c918c1-istio-podinfo" pod "675a6c4e-5415-4741-b381-257fd4c918c1" (UID: "675a6c4e-5415-4741-b381-257fd4c918c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:35:55.128268 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.128250    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-envoy" pod "675a6c4e-5415-4741-b381-257fd4c918c1" (UID: "675a6c4e-5415-4741-b381-257fd4c918c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:35:55.180793 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.180761    4484 generic.go:356] "Generic (PLEG): container finished" podID="675a6c4e-5415-4741-b381-257fd4c918c1" containerID="1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67" exitCode=0
Oct 31 18:35:55.181170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.180827    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"
Oct 31 18:35:55.181170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.180830    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" event={"ID":"675a6c4e-5415-4741-b381-257fd4c918c1","Type":"ContainerDied","Data":"1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67"}
Oct 31 18:35:55.181170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.180889    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt" event={"ID":"675a6c4e-5415-4741-b381-257fd4c918c1","Type":"ContainerDied","Data":"0565c0bab6d4256298673adfa7c5639b8f654e514d321e323919e972b198c4d1"}
Oct 31 18:35:55.181170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.180920    4484 scope.go:122] "RemoveContainer" containerID="1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67"
Oct 31 18:35:55.198078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.198052    4484 scope.go:122] "RemoveContainer" containerID="1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67"
Oct 31 18:35:55.198410 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:35:55.198385    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67\": container with ID starting with 1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67 not found: ID does not exist" containerID="1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67"
Oct 31 18:35:55.198501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.198423    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67"} err="failed to get container status \"1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67\": rpc error: code = NotFound desc = could not find container \"1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67\": container with ID starting with 1c7576f7eb521b8ee64cd38ed09b1b05f628444a659171250df8bb64a7e56d67 not found: ID does not exist"
Oct 31 18:35:55.226567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.226539    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.226567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.226564    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.226760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.226578    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdpzn\" (UniqueName: \"kubernetes.io/projected/675a6c4e-5415-4741-b381-257fd4c918c1-kube-api-access-bdpzn\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.226760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.226589    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/675a6c4e-5415-4741-b381-257fd4c918c1-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.226760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.226598    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/675a6c4e-5415-4741-b381-257fd4c918c1-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:35:55.347484 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.347427    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"]
Oct 31 18:35:55.352710 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.352668    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4xgvt"]
Oct 31 18:35:55.468887 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:35:55.468802    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:35:55.993553 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:55.993514    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" path="/var/lib/kubelet/pods/675a6c4e-5415-4741-b381-257fd4c918c1/volumes"
Oct 31 18:35:57.851353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:35:57.851322    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:02.568006 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:02.567966    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:05.570963 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.570931    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"]
Oct 31 18:36:05.571424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.571413    4484 cpu_manager.go:414] "containerMap: removing container" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" containerName="istio-proxy"
Oct 31 18:36:05.571465 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.571424    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" containerName="istio-proxy"
Oct 31 18:36:05.571518 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.571509    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="675a6c4e-5415-4741-b381-257fd4c918c1" containerName="istio-proxy"
Oct 31 18:36:05.576032 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.576017    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:05.630973 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.630938    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"]
Oct 31 18:36:05.678663 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.678629    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:36:05.678663 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.678665    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:36:05.737912 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.737862    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b82135e3-cqq8k\" (UID: \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:05.738111 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.737935    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b82135e3-cqq8k\" (UID: \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:05.738111 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.738033    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gfpw\" (UniqueName: \"kubernetes.io/projected/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-kube-api-access-2gfpw\") pod \"maas-api-revoke-sub-b82135e3-cqq8k\" (UID: \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:05.838808 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.838709    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b82135e3-cqq8k\" (UID: \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:05.838808 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.838772    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b82135e3-cqq8k\" (UID: \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:05.839049 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.838822    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gfpw\" (UniqueName: \"kubernetes.io/projected/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-kube-api-access-2gfpw\") pod \"maas-api-revoke-sub-b82135e3-cqq8k\" (UID: \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:05.839533 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.839510    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b82135e3-cqq8k\" (UID: \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:05.841300 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.841280    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b82135e3-cqq8k\" (UID: \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:05.897121 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:05.897086    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gfpw\" (UniqueName: \"kubernetes.io/projected/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-kube-api-access-2gfpw\") pod \"maas-api-revoke-sub-b82135e3-cqq8k\" (UID: \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:06.188263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.188185    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:06.560576 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.560548    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"]
Oct 31 18:36:06.562556 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:36:06.562527    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ae0eac_eaeb_4a7e_92fe_0606deee7ff2.slice/crio-a6fea769946246c853f5c4cc2c6b0c1deb559ef1733f1671fbaaa1c06eb6d754 WatchSource:0}: Error finding container a6fea769946246c853f5c4cc2c6b0c1deb559ef1733f1671fbaaa1c06eb6d754: Status 404 returned error can't find the container with id a6fea769946246c853f5c4cc2c6b0c1deb559ef1733f1671fbaaa1c06eb6d754
Oct 31 18:36:06.680230 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.680191    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"]
Oct 31 18:36:06.685491 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.685450    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.738214 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.738184    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3e4a2c-openshift-default-dockercfg-7jr6n\""
Oct 31 18:36:06.784348 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.784317    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"]
Oct 31 18:36:06.850713 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.850637    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4cm\" (UniqueName: \"kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-kube-api-access-vz4cm\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.850713 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.850673    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.850888 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.850799    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97d19fe-d536-42cd-8529-39d26421d668-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.850888 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.850832    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97d19fe-d536-42cd-8529-39d26421d668-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.850888 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.850860    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.850984 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.850904    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.851017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.850985    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.851049 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.851027    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.851122 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.851105    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.952514 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.952458    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4cm\" (UniqueName: \"kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-kube-api-access-vz4cm\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.952514 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.952516    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.952775 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.952547    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97d19fe-d536-42cd-8529-39d26421d668-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.952775 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.952562    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97d19fe-d536-42cd-8529-39d26421d668-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.952775 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.952583    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.952775 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.952627    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.952775 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.952678    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.952775 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.952709    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.952775 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.952753    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.953096 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.953075    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.953170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.953145    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.953300 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.953222    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.953300 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.953264    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.953539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.953521    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97d19fe-d536-42cd-8529-39d26421d668-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.955129 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.955111    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97d19fe-d536-42cd-8529-39d26421d668-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:06.955214 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:06.955136    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:07.012324 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:07.012292    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:07.012453 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:07.012347    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4cm\" (UniqueName: \"kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-kube-api-access-vz4cm\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:07.253465 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:07.253421    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k" event={"ID":"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2","Type":"ContainerStarted","Data":"5ae53c5f88c841f4176864534cf08c6144c6500cab86897489a2c2bee3d4cbec"}
Oct 31 18:36:07.253465 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:07.253493    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k" event={"ID":"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2","Type":"ContainerStarted","Data":"a6fea769946246c853f5c4cc2c6b0c1deb559ef1733f1671fbaaa1c06eb6d754"}
Oct 31 18:36:07.254361 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:07.254333    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:07.298899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:07.298861    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:07.368255 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:07.368206    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k" podStartSLOduration=2.368190066 podStartE2EDuration="2.368190066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:36:07.367174742 +0000 UTC m=+913.513624607" watchObservedRunningTime="2026-10-31 18:36:07.368190066 +0000 UTC m=+913.514639929"
Oct 31 18:36:07.723528 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:07.723501    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"]
Oct 31 18:36:07.724710 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:36:07.724684    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97d19fe_d536_42cd_8529_39d26421d668.slice/crio-cedc8cb6d313fdfe269c0ec2658d80b71b3746b3516d10aeb7de54ea9ca42c8d WatchSource:0}: Error finding container cedc8cb6d313fdfe269c0ec2658d80b71b3746b3516d10aeb7de54ea9ca42c8d: Status 404 returned error can't find the container with id cedc8cb6d313fdfe269c0ec2658d80b71b3746b3516d10aeb7de54ea9ca42c8d
Oct 31 18:36:07.727039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:07.727007    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:36:07.727137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:07.727083    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:36:07.727137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:07.727126    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:36:07.875549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:07.875514    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:36:08.287335 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:08.287302    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6" event={"ID":"d97d19fe-d536-42cd-8529-39d26421d668","Type":"ContainerStarted","Data":"c54bc2f2469f85bcacd086d689e922083d3b14f4082faa6a9fb19e862d09e53c"}
Oct 31 18:36:08.287570 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:08.287367    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6" event={"ID":"d97d19fe-d536-42cd-8529-39d26421d668","Type":"ContainerStarted","Data":"cedc8cb6d313fdfe269c0ec2658d80b71b3746b3516d10aeb7de54ea9ca42c8d"}
Oct 31 18:36:08.289330 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:08.289303    4484 generic.go:356] "Generic (PLEG): container finished" podID="30ae0eac-eaeb-4a7e-92fe-0606deee7ff2" containerID="5ae53c5f88c841f4176864534cf08c6144c6500cab86897489a2c2bee3d4cbec" exitCode=0
Oct 31 18:36:08.289463 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:08.289385    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k" event={"ID":"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2","Type":"ContainerDied","Data":"5ae53c5f88c841f4176864534cf08c6144c6500cab86897489a2c2bee3d4cbec"}
Oct 31 18:36:08.299879 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:08.299850    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:08.301713 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:08.301678    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 18:36:08.301864 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:08.301733    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6" podUID="d97d19fe-d536-42cd-8529-39d26421d668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 18:36:08.407216 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:08.407163    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6" podStartSLOduration=2.407149261 podStartE2EDuration="2.407149261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:36:08.404922889 +0000 UTC m=+914.551372753" watchObservedRunningTime="2026-10-31 18:36:08.407149261 +0000 UTC m=+914.553599194"
Oct 31 18:36:09.299284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:09.299260    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 18:36:09.299657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:09.299310    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6" podUID="d97d19fe-d536-42cd-8529-39d26421d668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 18:36:09.446863 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:09.446842    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:09.500366 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:09.500330    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-kube-api-access-2gfpw\" (UniqueName: \"kubernetes.io/projected/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-kube-api-access-2gfpw\") pod \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\" (UID: \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\") "
Oct 31 18:36:09.500570 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:09.500450    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-openshift-ca\") pod \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\" (UID: \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\") "
Oct 31 18:36:09.500570 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:09.500516    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-xks-ca\") pod \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\" (UID: \"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2\") "
Oct 31 18:36:09.500770 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:09.500749    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-openshift-ca" pod "30ae0eac-eaeb-4a7e-92fe-0606deee7ff2" (UID: "30ae0eac-eaeb-4a7e-92fe-0606deee7ff2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:36:09.500909 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:09.500877    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:36:09.502695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:09.502676    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-kube-api-access-2gfpw" pod "30ae0eac-eaeb-4a7e-92fe-0606deee7ff2" (UID: "30ae0eac-eaeb-4a7e-92fe-0606deee7ff2"). InnerVolumeSpecName "kube-api-access-2gfpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:36:09.502798 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:09.502778    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-xks-ca" pod "30ae0eac-eaeb-4a7e-92fe-0606deee7ff2" (UID: "30ae0eac-eaeb-4a7e-92fe-0606deee7ff2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:36:09.601726 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:09.601626    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:36:09.601726 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:09.601656    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gfpw\" (UniqueName: \"kubernetes.io/projected/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2-kube-api-access-2gfpw\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:36:10.299628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:10.299542    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 18:36:10.299628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:10.299594    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6" podUID="d97d19fe-d536-42cd-8529-39d26421d668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 18:36:10.301548 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:10.301512    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k" event={"ID":"30ae0eac-eaeb-4a7e-92fe-0606deee7ff2","Type":"ContainerDied","Data":"a6fea769946246c853f5c4cc2c6b0c1deb559ef1733f1671fbaaa1c06eb6d754"}
Oct 31 18:36:10.301668 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:10.301583    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6fea769946246c853f5c4cc2c6b0c1deb559ef1733f1671fbaaa1c06eb6d754"
Oct 31 18:36:10.301668 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:10.301534    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"
Oct 31 18:36:11.299527 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:11.299468    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 18:36:11.299740 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:11.299558    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6" podUID="d97d19fe-d536-42cd-8529-39d26421d668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 18:36:11.982928 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:11.982887    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:12.299216 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:12.299180    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 18:36:12.299423 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:12.299240    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6" podUID="d97d19fe-d536-42cd-8529-39d26421d668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 18:36:13.299671 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:13.299632    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 18:36:13.300167 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:13.299696    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6" podUID="d97d19fe-d536-42cd-8529-39d26421d668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 18:36:14.915343 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:14.915303    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:14.915914 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:14.915835    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:14.916801 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:14.916779    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 18:36:16.651514 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:16.651461    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:18.833593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.833558    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"]
Oct 31 18:36:18.834129 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.834115    4484 cpu_manager.go:414] "containerMap: removing container" podUID="30ae0eac-eaeb-4a7e-92fe-0606deee7ff2" containerName="revoke-keys"
Oct 31 18:36:18.834172 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.834131    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ae0eac-eaeb-4a7e-92fe-0606deee7ff2" containerName="revoke-keys"
Oct 31 18:36:18.834231 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.834220    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ae0eac-eaeb-4a7e-92fe-0606deee7ff2" containerName="revoke-keys"
Oct 31 18:36:18.837824 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.837802    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:18.890700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.890665    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a8587d-openshift-default-dockercfg-zdp6b\""
Oct 31 18:36:18.905199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.905166    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:18.905199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.905198    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:18.905434 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.905218    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:18.905525 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.905430    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:18.905525 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.905504    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:18.905636 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.905544    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:18.905690 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.905641    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:18.905690 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.905676    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbfcz\" (UniqueName: \"kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-kube-api-access-dbfcz\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:18.905771 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.905707    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:18.945159 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:18.945124    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"]
Oct 31 18:36:19.006575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.006540    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.006575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.006577    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.006828 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.006598    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.006828 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.006630    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.006828 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.006648    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbfcz\" (UniqueName: \"kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-kube-api-access-dbfcz\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.006828 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.006666    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.006828 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.006689    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.006828 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.006794    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.007119 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.006865    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.007119 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.006966    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.007119 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.007018    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.007119 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.007079    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.007323 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.007259    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.007458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.007437    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.009080 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.009060    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.009167 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.009092    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.064716 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.064677    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.064886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.064796    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbfcz\" (UniqueName: \"kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-kube-api-access-dbfcz\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.151721 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.151630    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:19.579900 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.579875    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"]
Oct 31 18:36:19.580653 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:36:19.580630    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ee3c68_0043_49ac_be93_fccb0cb0f932.slice/crio-d575ea0463b1026d9255002a41fd2714169a3f343b403a0fd68b46a3bdda8002 WatchSource:0}: Error finding container d575ea0463b1026d9255002a41fd2714169a3f343b403a0fd68b46a3bdda8002: Status 404 returned error can't find the container with id d575ea0463b1026d9255002a41fd2714169a3f343b403a0fd68b46a3bdda8002
Oct 31 18:36:19.583011 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.582978    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:36:19.583097 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.583046    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:36:19.583097 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:19.583075    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:36:20.385841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:20.385796    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq" event={"ID":"c3ee3c68-0043-49ac-be93-fccb0cb0f932","Type":"ContainerStarted","Data":"e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0"}
Oct 31 18:36:20.386500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:20.386441    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq" event={"ID":"c3ee3c68-0043-49ac-be93-fccb0cb0f932","Type":"ContainerStarted","Data":"d575ea0463b1026d9255002a41fd2714169a3f343b403a0fd68b46a3bdda8002"}
Oct 31 18:36:20.560375 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:20.560318    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq" podStartSLOduration=2.5603035690000002 podStartE2EDuration="2.560303569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:36:20.557873545 +0000 UTC m=+926.704323409" watchObservedRunningTime="2026-10-31 18:36:20.560303569 +0000 UTC m=+926.706753433"
Oct 31 18:36:21.152096 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:21.152013    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:21.153971 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:21.153940    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 18:36:21.154141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:21.154000    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq" podUID="c3ee3c68-0043-49ac-be93-fccb0cb0f932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 18:36:21.455870 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:21.455770    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:22.152162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:22.152126    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 18:36:22.152342 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:22.152182    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq" podUID="c3ee3c68-0043-49ac-be93-fccb0cb0f932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 18:36:23.152746 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:23.152714    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 18:36:23.153118 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:23.152774    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq" podUID="c3ee3c68-0043-49ac-be93-fccb0cb0f932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 18:36:24.152027 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:24.151990    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 18:36:24.152230 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:24.152057    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq" podUID="c3ee3c68-0043-49ac-be93-fccb0cb0f932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 18:36:25.152774 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:25.152741    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 31 18:36:25.153150 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:25.152790    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq" podUID="c3ee3c68-0043-49ac-be93-fccb0cb0f932" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 31 18:36:25.493851 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:36:25.493818    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:36:26.147337 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:26.147289    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:26.924341 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:26.924309    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:26.925170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:26.925146    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:26.926626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:26.926598    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 18:36:30.848245 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:30.848213    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:33.700215 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:33.700177    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:33.909775 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:33.909734    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:35.348482 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:35.348433    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:36:35.570796 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:35.570759    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:35.730567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:35.730439    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:35.891292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:35.891256    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:36.039408 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:36.039367    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:40.251657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:40.251621    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:44.583972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:44.583939    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:44.954910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:44.954821    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:45.359348 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:45.359319    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:49.764106 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:49.764075    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:50.065047 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:50.064963    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:54.213293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:54.213255    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:54.540889 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:54.540852    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:36:55.467248 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:36:55.467216    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:36:59.410703 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:36:59.410666    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:01.795797 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:01.795761    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:37:04.374378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:04.374331    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:05.350197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.350162    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"]
Oct 31 18:37:05.355458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.355438    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.400804 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.400775    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"]
Oct 31 18:37:05.404641 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.404627    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.451546 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.451504    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"]
Oct 31 18:37:05.499109 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.499083    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"]
Oct 31 18:37:05.516430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516404    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76199531-8025-41d1-813a-cc1a78125e06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.516568 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516439    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.516568 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516458    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.516568 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516506    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w85cv\" (UniqueName: \"kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-kube-api-access-w85cv\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.516672 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516621    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76199531-8025-41d1-813a-cc1a78125e06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.516672 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516659    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.516732 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516675    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.516769 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516750    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.516800 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516786    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.516835 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516801    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.516835 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516828    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.516897 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516881    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.516929 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516917    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.516966 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.516952    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.517026 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.517011    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.517060 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.517040    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.517095 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.517072    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpdl6\" (UniqueName: \"kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-kube-api-access-bpdl6\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.517172 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.517160    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.618833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.618732    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w85cv\" (UniqueName: \"kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-kube-api-access-w85cv\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.618833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.618788    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76199531-8025-41d1-813a-cc1a78125e06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.618833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.618822    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.619141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.618840    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.619141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.618871    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.619141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.618998    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.619141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619027    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.619141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619065    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.619141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619090    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.619141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619113    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.619141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619138    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.619589 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619184    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.619589 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619205    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.619589 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619245    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpdl6\" (UniqueName: \"kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-kube-api-access-bpdl6\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.619589 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619277    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.619589 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619325    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.619589 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619358    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76199531-8025-41d1-813a-cc1a78125e06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.619589 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619388    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.619589 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619413    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.619589 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619446    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.620040 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619593    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.620040 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619883    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.620040 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.619977    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.620192 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.620147    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.620192 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.620165    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.620301 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.620251    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.620819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.620792    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76199531-8025-41d1-813a-cc1a78125e06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.620917 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.620895    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.621716 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.621690    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.621979 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.621960    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.622295 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.622272    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.622422 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.622407    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76199531-8025-41d1-813a-cc1a78125e06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.676582 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.676542    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.676582 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.676543    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.676783 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.676542    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w85cv\" (UniqueName: \"kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-kube-api-access-w85cv\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wxkqh\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.676855 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.676831    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpdl6\" (UniqueName: \"kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-kube-api-access-bpdl6\") pod \"maas-default-gateway-openshift-default-7f694df8bc-hz57f\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:05.717361 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.717329    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:05.968660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:05.968575    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:06.142761 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.142737    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"]
Oct 31 18:37:06.143400 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:37:06.143371    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab39de0_0e3b_489f_91b9_3703f4957aa2.slice/crio-bb731c1535c60142637e13ab13d6380aad03d39bb2758944d9f7a361288073ee WatchSource:0}: Error finding container bb731c1535c60142637e13ab13d6380aad03d39bb2758944d9f7a361288073ee: Status 404 returned error can't find the container with id bb731c1535c60142637e13ab13d6380aad03d39bb2758944d9f7a361288073ee
Oct 31 18:37:06.145732 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.145700    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:37:06.145815 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.145762    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:37:06.145815 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.145789    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:37:06.342579 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.342550    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"]
Oct 31 18:37:06.343272 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:37:06.343244    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76199531_8025_41d1_813a_cc1a78125e06.slice/crio-a329bd3a062d03c68a4570acad848285fbf42cc04d0b92285f335f1d102deb1d WatchSource:0}: Error finding container a329bd3a062d03c68a4570acad848285fbf42cc04d0b92285f335f1d102deb1d: Status 404 returned error can't find the container with id a329bd3a062d03c68a4570acad848285fbf42cc04d0b92285f335f1d102deb1d
Oct 31 18:37:06.345783 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.345750    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:37:06.345884 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.345830    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:37:06.345884 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.345878    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:37:06.656575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.656407    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" event={"ID":"76199531-8025-41d1-813a-cc1a78125e06","Type":"ContainerStarted","Data":"925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723"}
Oct 31 18:37:06.658174 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.657072    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" event={"ID":"76199531-8025-41d1-813a-cc1a78125e06","Type":"ContainerStarted","Data":"a329bd3a062d03c68a4570acad848285fbf42cc04d0b92285f335f1d102deb1d"}
Oct 31 18:37:06.660495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.660439    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" event={"ID":"3ab39de0-0e3b-489f-91b9-3703f4957aa2","Type":"ContainerStarted","Data":"e2b317cae9f74f8b500dfc898e7dd599533c936ad970c57c6019a1180112f52c"}
Oct 31 18:37:06.662350 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.661650    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" event={"ID":"3ab39de0-0e3b-489f-91b9-3703f4957aa2","Type":"ContainerStarted","Data":"bb731c1535c60142637e13ab13d6380aad03d39bb2758944d9f7a361288073ee"}
Oct 31 18:37:06.718564 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.718532    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:06.904049 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.903835    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" podStartSLOduration=1.903814691 podStartE2EDuration="1.903814691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:37:06.78596052 +0000 UTC m=+972.932410384" watchObservedRunningTime="2026-10-31 18:37:06.903814691 +0000 UTC m=+973.050264558"
Oct 31 18:37:06.905722 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.905674    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" podStartSLOduration=1.905662037 podStartE2EDuration="1.905662037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:37:06.902937021 +0000 UTC m=+973.049386885" watchObservedRunningTime="2026-10-31 18:37:06.905662037 +0000 UTC m=+973.052111901"
Oct 31 18:37:06.982561 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:06.977578    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:07.731535 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:07.726539    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wxkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 18:37:07.731535 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:07.726602    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" podUID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 18:37:07.980640 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:07.980602    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 18:37:07.980932 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:07.980901    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" podUID="76199531-8025-41d1-813a-cc1a78125e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 18:37:08.718001 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:08.717966    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wxkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 18:37:08.718191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:08.718013    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" podUID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 18:37:08.985568 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:08.984979    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 18:37:08.985568 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:08.985039    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" podUID="76199531-8025-41d1-813a-cc1a78125e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 18:37:09.220500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:09.198822    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:09.718226 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:09.718189    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wxkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 18:37:09.718402 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:09.718248    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" podUID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 18:37:09.975556 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:09.973284    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 18:37:09.975556 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:09.973341    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" podUID="76199531-8025-41d1-813a-cc1a78125e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 18:37:10.718661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:10.718602    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wxkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 18:37:10.719154 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:10.718674    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" podUID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 18:37:10.993513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:10.993445    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 18:37:10.993807 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:10.993780    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" podUID="76199531-8025-41d1-813a-cc1a78125e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 18:37:11.718857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:11.718821    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wxkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 18:37:11.719362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:11.718886    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" podUID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 18:37:11.983690 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:11.983598    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 18:37:11.983956 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:11.983930    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" podUID="76199531-8025-41d1-813a-cc1a78125e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 18:37:12.718916 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:12.718880    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wxkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 18:37:12.719572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:12.719539    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" podUID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 18:37:12.969687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:12.969583    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz57f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 18:37:12.969687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:12.969636    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" podUID="76199531-8025-41d1-813a-cc1a78125e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 18:37:13.986702 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:13.986662    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:14.578304 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:14.578269    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:14.578699 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:14.578669    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:14.579640 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:14.579611    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:37:14.707012 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:14.706983    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:14.707298 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:14.707276    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:14.708310 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:14.708287    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:37:18.654356 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:18.654318    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:23.411939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:23.411906    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:25.467303 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:37:25.467258    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:37:28.162729 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:28.162693    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:32.966407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:32.966362    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:37.307333 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.307291    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"]
Oct 31 18:37:37.311994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.311975    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.367779 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.367748    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"]
Oct 31 18:37:37.414238 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.414201    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3e4a2c\"/\"distinct-w3-3e4a2c-kserve-self-signed-certs\""
Oct 31 18:37:37.423376 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.423348    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fdc08a1-c720-4038-acca-b931fbec7633-tls-certs\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.423376 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.423380    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-tmp-dir\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.423635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.423403    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-home\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.423635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.423543    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwt25\" (UniqueName: \"kubernetes.io/projected/7fdc08a1-c720-4038-acca-b931fbec7633-kube-api-access-vwt25\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.423635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.423570    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-dshm\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.423635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.423598    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-model-cache\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.437346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.437306    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:37.525131 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.525101    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fdc08a1-c720-4038-acca-b931fbec7633-tls-certs\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.525131 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.525134    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-tmp-dir\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.525392 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.525156    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-home\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.525392 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.525205    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwt25\" (UniqueName: \"kubernetes.io/projected/7fdc08a1-c720-4038-acca-b931fbec7633-kube-api-access-vwt25\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.525392 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.525225    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-dshm\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.525392 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.525247    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-model-cache\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.525652 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.525617    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-home\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.525707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.525680    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-tmp-dir\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.525744 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.525717    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-model-cache\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.527718 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.527698    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-dshm\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.527881 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.527862    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fdc08a1-c720-4038-acca-b931fbec7633-tls-certs\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.584038 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.583963    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwt25\" (UniqueName: \"kubernetes.io/projected/7fdc08a1-c720-4038-acca-b931fbec7633-kube-api-access-vwt25\") pod \"distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.626498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.626457    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:37.860167 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:37.860079    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:38.002924 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:38.002899    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"]
Oct 31 18:37:38.004145 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:37:38.004123    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fdc08a1_c720_4038_acca_b931fbec7633.slice/crio-b2264dded3c0f822be5db183dcc8943afc982a98ad53dcb9cb30f84c08d6a80e WatchSource:0}: Error finding container b2264dded3c0f822be5db183dcc8943afc982a98ad53dcb9cb30f84c08d6a80e: Status 404 returned error can't find the container with id b2264dded3c0f822be5db183dcc8943afc982a98ad53dcb9cb30f84c08d6a80e
Oct 31 18:37:38.941489 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:38.941435    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4" event={"ID":"7fdc08a1-c720-4038-acca-b931fbec7633","Type":"ContainerStarted","Data":"88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a"}
Oct 31 18:37:38.941854 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:38.941500    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4" event={"ID":"7fdc08a1-c720-4038-acca-b931fbec7633","Type":"ContainerStarted","Data":"b2264dded3c0f822be5db183dcc8943afc982a98ad53dcb9cb30f84c08d6a80e"}
Oct 31 18:37:39.062526 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:39.062451    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4" podStartSLOduration=2.062436501 podStartE2EDuration="2.062436501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:37:39.060902299 +0000 UTC m=+1005.207352164" watchObservedRunningTime="2026-10-31 18:37:39.062436501 +0000 UTC m=+1005.208886365"
Oct 31 18:37:39.627353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:39.627318    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:42.465335 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:42.465296    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:44.553786 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:44.553751    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:37:45.767994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:45.767958    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:47.632503 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:47.631567    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:47.645710 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:47.645688    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:47.703136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:47.703047    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:48.036417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:48.036383    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:37:48.271102 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:48.271066    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:48.435390 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:48.435299    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:50.400050 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:50.400018    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:52.387411 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:52.387380    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:37:53.054269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:53.054235    4484 generic.go:356] "Generic (PLEG): container finished" podID="ea33ee05-e391-4d64-a57a-039b32d83cd2" containerID="1d38493a99a1736c5921b2e4d3800a1c02eb763fb9added986643df1dbb9ea9b" exitCode=1
Oct 31 18:37:53.054439 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:53.054288    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k" event={"ID":"ea33ee05-e391-4d64-a57a-039b32d83cd2","Type":"ContainerDied","Data":"1d38493a99a1736c5921b2e4d3800a1c02eb763fb9added986643df1dbb9ea9b"}
Oct 31 18:37:53.054701 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:53.054686    4484 scope.go:122] "RemoveContainer" containerID="1d38493a99a1736c5921b2e4d3800a1c02eb763fb9added986643df1dbb9ea9b"
Oct 31 18:37:53.055137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:53.055124    4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:37:53.978384 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:53.978349    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 18:37:54.063456 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:54.063424    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k" event={"ID":"ea33ee05-e391-4d64-a57a-039b32d83cd2","Type":"ContainerStarted","Data":"d78bbd34b6b18d47eda1613cea4b953b3f8d400586fa76222863ef37176ee3db"}
Oct 31 18:37:54.063652 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:54.063633    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"
Oct 31 18:37:55.468761 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:37:55.468724    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:37:57.083131 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:37:57.083099    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:02.063660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:02.063626    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:04.070850 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.070814    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bcvft-hzf9d"]
Oct 31 18:38:04.074918 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.074901    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.179789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.179752    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 18:38:04.179789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.179769    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6wm76\""
Oct 31 18:38:04.182214 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.182194    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bcvft-hzf9d"]
Oct 31 18:38:04.236526 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.236497    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlk5m\" (UniqueName: \"kubernetes.io/projected/4ab9d0e3-e0df-49d0-b281-d91156fad243-kube-api-access-dlk5m\") pod \"periodic-gathering-bcvft-hzf9d\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") " pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.236701 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.236539    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-service-ca-bundle\") pod \"periodic-gathering-bcvft-hzf9d\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") " pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.236701 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.236572    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-trusted-ca-bundle\") pod \"periodic-gathering-bcvft-hzf9d\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") " pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.236777 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.236753    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ab9d0e3-e0df-49d0-b281-d91156fad243-archives-path\") pod \"periodic-gathering-bcvft-hzf9d\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") " pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.282236 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.282191    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 18:38:04.337520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.337411    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlk5m\" (UniqueName: \"kubernetes.io/projected/4ab9d0e3-e0df-49d0-b281-d91156fad243-kube-api-access-dlk5m\") pod \"periodic-gathering-bcvft-hzf9d\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") " pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.337520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.337485    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-service-ca-bundle\") pod \"periodic-gathering-bcvft-hzf9d\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") " pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.337520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.337517    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-trusted-ca-bundle\") pod \"periodic-gathering-bcvft-hzf9d\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") " pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.337803 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.337606    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ab9d0e3-e0df-49d0-b281-d91156fad243-archives-path\") pod \"periodic-gathering-bcvft-hzf9d\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") " pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.338043 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.338018    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ab9d0e3-e0df-49d0-b281-d91156fad243-archives-path\") pod \"periodic-gathering-bcvft-hzf9d\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") " pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.338267 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.338248    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-service-ca-bundle\") pod \"periodic-gathering-bcvft-hzf9d\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") " pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.338446 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.338427    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-trusted-ca-bundle\") pod \"periodic-gathering-bcvft-hzf9d\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") " pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.395689 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.395660    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlk5m\" (UniqueName: \"kubernetes.io/projected/4ab9d0e3-e0df-49d0-b281-d91156fad243-kube-api-access-dlk5m\") pod \"periodic-gathering-bcvft-hzf9d\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") " pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:04.687632 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:04.687536    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:38:05.064791 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:05.064765    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bcvft-hzf9d"]
Oct 31 18:38:05.065861 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:38:05.065839    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab9d0e3_e0df_49d0_b281_d91156fad243.slice/crio-078604ab5ef87ad3b27dfe2a925fefccec900b4ae362f782c9519ce426512e41 WatchSource:0}: Error finding container 078604ab5ef87ad3b27dfe2a925fefccec900b4ae362f782c9519ce426512e41: Status 404 returned error can't find the container with id 078604ab5ef87ad3b27dfe2a925fefccec900b4ae362f782c9519ce426512e41
Oct 31 18:38:05.071487 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:05.071457    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-c8b7d8cc9-rzk2k"
Oct 31 18:38:05.131025 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:05.131000    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bcvft-hzf9d" event={"ID":"4ab9d0e3-e0df-49d0-b281-d91156fad243","Type":"ContainerStarted","Data":"078604ab5ef87ad3b27dfe2a925fefccec900b4ae362f782c9519ce426512e41"}
Oct 31 18:38:05.552048 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:05.552011    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:38:06.139494 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:06.139433    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bcvft-hzf9d" event={"ID":"4ab9d0e3-e0df-49d0-b281-d91156fad243","Type":"ContainerStarted","Data":"19acab58e86d7784f0796933b14dafd4fee17f655a1f1b1e49ae9d1e86145016"}
Oct 31 18:38:06.256518 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:06.256444    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bcvft-hzf9d" podStartSLOduration=2.256431542 podStartE2EDuration="2.256431542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:38:06.255072098 +0000 UTC m=+1032.401521963" watchObservedRunningTime="2026-10-31 18:38:06.256431542 +0000 UTC m=+1032.402881406"
Oct 31 18:38:06.759963 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:06.759929    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:10.982255 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:10.982208    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"]
Oct 31 18:38:10.986351 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:10.986323    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.042546 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.042516    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"]
Oct 31 18:38:11.090157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.090125    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"openshift-service-ca.crt\""
Oct 31 18:38:11.090345 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.090213    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"kube-root-ca.crt\""
Oct 31 18:38:11.090406 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.090362    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"default-dockercfg-5cdvq\""
Oct 31 18:38:11.090546 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.090523    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-88085e\"/\"e2e-conflict-b-1a3e1910-kserve-self-signed-certs\""
Oct 31 18:38:11.090822 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.090795    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:11.107939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.107917    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-home\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.108037 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.107944    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-model-cache\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.108087 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.108039    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-dshm\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.108087 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.108082    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzl9g\" (UniqueName: \"kubernetes.io/projected/eec4901f-617d-482d-a4df-143749c1eeb7-kube-api-access-dzl9g\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.108167 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.108106    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec4901f-617d-482d-a4df-143749c1eeb7-tls-certs\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.108256 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.108241    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-tmp-dir\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.209000 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.208957    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-tmp-dir\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.209198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.209028    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-home\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.209198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.209055    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-model-cache\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.209198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.209122    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-dshm\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.209198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.209161    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzl9g\" (UniqueName: \"kubernetes.io/projected/eec4901f-617d-482d-a4df-143749c1eeb7-kube-api-access-dzl9g\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.209198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.209190    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec4901f-617d-482d-a4df-143749c1eeb7-tls-certs\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.209504 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.209441    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-tmp-dir\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.209577 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.209459    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-home\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.209577 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.209545    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-model-cache\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.211734 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.211712    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-dshm\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.211843 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.211803    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec4901f-617d-482d-a4df-143749c1eeb7-tls-certs\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.266306 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.266274    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzl9g\" (UniqueName: \"kubernetes.io/projected/eec4901f-617d-482d-a4df-143749c1eeb7-kube-api-access-dzl9g\") pod \"e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") " pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.298683 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.298649    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:11.442307 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.442275    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:11.682592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:11.682567    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"]
Oct 31 18:38:11.683074 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:38:11.683048    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeec4901f_617d_482d_a4df_143749c1eeb7.slice/crio-777dabe1ec5ff781209fbd0655bc410d40efd8a7a6fce7fedd8f7ad7b78d7783 WatchSource:0}: Error finding container 777dabe1ec5ff781209fbd0655bc410d40efd8a7a6fce7fedd8f7ad7b78d7783: Status 404 returned error can't find the container with id 777dabe1ec5ff781209fbd0655bc410d40efd8a7a6fce7fedd8f7ad7b78d7783
Oct 31 18:38:12.176819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:12.176773    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm" event={"ID":"eec4901f-617d-482d-a4df-143749c1eeb7","Type":"ContainerStarted","Data":"aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a"}
Oct 31 18:38:12.176819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:12.176836    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm" event={"ID":"eec4901f-617d-482d-a4df-143749c1eeb7","Type":"ContainerStarted","Data":"777dabe1ec5ff781209fbd0655bc410d40efd8a7a6fce7fedd8f7ad7b78d7783"}
Oct 31 18:38:12.293157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:12.293106    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm" podStartSLOduration=2.293091968 podStartE2EDuration="2.293091968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:38:12.292619755 +0000 UTC m=+1038.439069619" watchObservedRunningTime="2026-10-31 18:38:12.293091968 +0000 UTC m=+1038.439541825"
Oct 31 18:38:12.299306 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:12.299283    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:16.252620 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:16.252574    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:17.527362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:17.527323    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:17.689134 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:17.689091    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:17.848104 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:17.848022    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:20.994430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:20.994381    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:21.299242 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:21.299150    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:21.311958 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:21.311932    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:22.245763 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:22.245739    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:38:25.468318 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:38:25.468280    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:38:25.711714 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:25.711680    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:29.986752 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:29.986714    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:30.472265 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:30.472171    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:35.015966 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:35.015934    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:38:35.171663 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:35.171630    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:39.969615 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:39.969571    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:41.847680 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.847645    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"]
Oct 31 18:38:41.851620 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.851600    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.889135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.889101    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.889284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.889139    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.889284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.889158    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.889284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.889216    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.889393 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.889293    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.889393 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.889328    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.889456 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.889427    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.889576 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.889561    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq7ng\" (UniqueName: \"kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-kube-api-access-cq7ng\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.889611 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.889595    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.957588 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.957555    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"]
Oct 31 18:38:41.990882 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.990842    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.991083 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.990964    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq7ng\" (UniqueName: \"kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-kube-api-access-cq7ng\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.991083 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.991003    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.991083 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.991037    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.991083 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.991070    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.991305 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.991097    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.991305 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.991120    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.991305 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.991182    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.991305 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.991231    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.991678 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.991571    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.991803 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.991715    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.991803 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.991756    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.991912 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.991894    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.992000 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.991978    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.994415 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.994393    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:41.994756 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:41.994734    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:42.048544 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:42.048509    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:42.048544 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:42.048528    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq7ng\" (UniqueName: \"kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-kube-api-access-cq7ng\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:42.165575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:42.165461    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:42.593201 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:42.593170    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"]
Oct 31 18:38:42.594414 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:38:42.594392    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217d6077_722f_4e9a_b9ee_d3e68ce59fa1.slice/crio-e4ada3600a8360dad24dbe551ad06d48459063c14c1df2dd41872c5e4fb52cc9 WatchSource:0}: Error finding container e4ada3600a8360dad24dbe551ad06d48459063c14c1df2dd41872c5e4fb52cc9: Status 404 returned error can't find the container with id e4ada3600a8360dad24dbe551ad06d48459063c14c1df2dd41872c5e4fb52cc9
Oct 31 18:38:42.596947 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:42.596915    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:38:42.597018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:42.596982    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:38:42.597018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:42.597009    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:38:43.391184 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:43.391149    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" event={"ID":"217d6077-722f-4e9a-b9ee-d3e68ce59fa1","Type":"ContainerStarted","Data":"4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7"}
Oct 31 18:38:43.391679 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:43.391208    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" event={"ID":"217d6077-722f-4e9a-b9ee-d3e68ce59fa1","Type":"ContainerStarted","Data":"e4ada3600a8360dad24dbe551ad06d48459063c14c1df2dd41872c5e4fb52cc9"}
Oct 31 18:38:43.560318 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:43.560271    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" podStartSLOduration=2.560255009 podStartE2EDuration="2.560255009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:38:43.559524412 +0000 UTC m=+1069.705974276" watchObservedRunningTime="2026-10-31 18:38:43.560255009 +0000 UTC m=+1069.706704873"
Oct 31 18:38:44.179263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:44.174108    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:44.185997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:44.185606    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:38:44.185997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:44.185670    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 18:38:44.694281 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:44.694240    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:45.176512 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:45.176129    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:38:45.176512 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:45.176195    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 18:38:46.111919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:46.111882    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:46.173167 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:46.173127    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:38:46.173500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:46.173440    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 18:38:47.177498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:47.174071    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:38:47.177498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:47.174146    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 18:38:48.176183 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:48.176146    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:38:48.176955 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:48.176925    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 18:38:48.959897 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:48.959865    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"]
Oct 31 18:38:48.963983 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:48.963967    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.067781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.067748    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3e4a2c\"/\"distinct-2-w3-3e4a2c-kserve-self-signed-certs\""
Oct 31 18:38:49.070596 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.070568    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"]
Oct 31 18:38:49.071289 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.071258    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:49.080741 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.080715    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-tmp-dir\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.080874 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.080811    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-home\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.081188 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.081163    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/307a9ca2-cb52-4131-b750-2fb5d89e8235-tls-certs\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.081264 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.081218    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-dshm\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.081264 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.081251    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9dn\" (UniqueName: \"kubernetes.io/projected/307a9ca2-cb52-4131-b750-2fb5d89e8235-kube-api-access-cp9dn\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.081397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.081378    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-model-cache\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.166266 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.166232    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 18:38:49.166504 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.166290    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 18:38:49.183035 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.183001    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/307a9ca2-cb52-4131-b750-2fb5d89e8235-tls-certs\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.183231 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.183053    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-dshm\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.183231 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.183076    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9dn\" (UniqueName: \"kubernetes.io/projected/307a9ca2-cb52-4131-b750-2fb5d89e8235-kube-api-access-cp9dn\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.183231 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.183097    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-model-cache\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.183231 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.183133    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-tmp-dir\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.183462 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.183308    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-home\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.183577 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.183557    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-tmp-dir\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.183629 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.183559    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-model-cache\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.183750 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.183728    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-home\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.185550 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.185525    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-dshm\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.185697 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.185681    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/307a9ca2-cb52-4131-b750-2fb5d89e8235-tls-certs\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.240956 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.240918    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9dn\" (UniqueName: \"kubernetes.io/projected/307a9ca2-cb52-4131-b750-2fb5d89e8235-kube-api-access-cp9dn\") pod \"distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") " pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.277652 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.277613    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:49.474205 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.474174    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:49.668653 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:49.668626    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"]
Oct 31 18:38:49.670233 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:38:49.670203    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod307a9ca2_cb52_4131_b750_2fb5d89e8235.slice/crio-2a4649f204f1aebefe54fbc3ee9d2fdb1a303eabc196077317756378f83434ee WatchSource:0}: Error finding container 2a4649f204f1aebefe54fbc3ee9d2fdb1a303eabc196077317756378f83434ee: Status 404 returned error can't find the container with id 2a4649f204f1aebefe54fbc3ee9d2fdb1a303eabc196077317756378f83434ee
Oct 31 18:38:50.170395 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:50.170363    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:50.170857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:50.170663    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:50.171411 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:50.171395    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:38:50.439208 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:50.439109    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c" event={"ID":"307a9ca2-cb52-4131-b750-2fb5d89e8235","Type":"ContainerStarted","Data":"d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1"}
Oct 31 18:38:50.439208 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:50.439157    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c" event={"ID":"307a9ca2-cb52-4131-b750-2fb5d89e8235","Type":"ContainerStarted","Data":"2a4649f204f1aebefe54fbc3ee9d2fdb1a303eabc196077317756378f83434ee"}
Oct 31 18:38:50.557440 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:50.557376    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c" podStartSLOduration=2.557357456 podStartE2EDuration="2.557357456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:38:50.555899282 +0000 UTC m=+1076.702349146" watchObservedRunningTime="2026-10-31 18:38:50.557357456 +0000 UTC m=+1076.703807322"
Oct 31 18:38:50.748550 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:50.748517    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:51.278015 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:51.277983    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:53.766381 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:53.766349    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:54.129280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:54.129206    4484 scope.go:122] "RemoveContainer" containerID="2b9177016cd950103e4c06584be4edf3873d088c197eec67264af084fddce1f3"
Oct 31 18:38:54.139406 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:54.139381    4484 scope.go:122] "RemoveContainer" containerID="3f24a335ff7bd4aeafd6940471ae5945023ef7e0392be422a9afe27a82aa0ca4"
Oct 31 18:38:54.148063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:54.148037    4484 scope.go:122] "RemoveContainer" containerID="7010cbf616913179405c33892241511d9246d6143f35c2afdcd77d25634a4c04"
Oct 31 18:38:54.282693 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:54.282655    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:55.468423 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:38:55.468383    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:38:58.980220 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:58.980187    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:59.140568 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:59.140518    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:38:59.278716 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:59.278675    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:59.293909 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:59.293882    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:38:59.501768 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:38:59.501736    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:39:03.725211 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:03.725179    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:08.353598 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:08.353551    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:09.918644 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:09.918605    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:13.142673 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:13.142641    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:13.418334 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:13.418256    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:39:17.857625 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:17.857587    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:22.563388 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:22.563350    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:23.196760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.196723    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"]
Oct 31 18:39:23.200845 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.200825    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.264787 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.264759    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.264963 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.264803    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.264963 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.264879    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.265074 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.264981    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.265074 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.265037    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.265157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.265096    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.265199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.265155    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb4ws\" (UniqueName: \"kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-kube-api-access-pb4ws\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.265311 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.265288    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.265452 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.265337    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.285052 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.285023    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"]
Oct 31 18:39:23.366336 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.366305    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.366574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.366360    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.366574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.366392    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.366574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.366420    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.366574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.366519    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.366574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.366566    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.366841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.366589    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.366841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.366621    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb4ws\" (UniqueName: \"kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-kube-api-access-pb4ws\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.366841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.366760    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.366841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.366791    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.367030 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.366947    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.367127 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.367104    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.367188 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.367119    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.367331 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.367310    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.369439 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.369415    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.370120 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.370098    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.425523 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.425492    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.425870 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.425849    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb4ws\" (UniqueName: \"kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-kube-api-access-pb4ws\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.513816 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.513791    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:23.941004 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.940976    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"]
Oct 31 18:39:23.941768 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:39:23.941728    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4eaec1_65e4_4507_b4fd_7938a9adcd87.slice/crio-c7547935f908653398c17424b7de5be7d6e6b40deba51b83aa9a79528d60ef04 WatchSource:0}: Error finding container c7547935f908653398c17424b7de5be7d6e6b40deba51b83aa9a79528d60ef04: Status 404 returned error can't find the container with id c7547935f908653398c17424b7de5be7d6e6b40deba51b83aa9a79528d60ef04
Oct 31 18:39:23.944142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.944109    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:39:23.944208 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.944177    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:39:23.944248 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:23.944211    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:39:24.631704 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:24.631671    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n" event={"ID":"ec4eaec1-65e4-4507-b4fd-7938a9adcd87","Type":"ContainerStarted","Data":"e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c"}
Oct 31 18:39:24.631867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:24.631748    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n" event={"ID":"ec4eaec1-65e4-4507-b4fd-7938a9adcd87","Type":"ContainerStarted","Data":"c7547935f908653398c17424b7de5be7d6e6b40deba51b83aa9a79528d60ef04"}
Oct 31 18:39:24.853872 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:24.853813    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n" podStartSLOduration=1.8537994549999999 podStartE2EDuration="1.853799455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:39:24.801763937 +0000 UTC m=+1110.948213800" watchObservedRunningTime="2026-10-31 18:39:24.853799455 +0000 UTC m=+1111.000249393"
Oct 31 18:39:25.492746 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:39:25.492706    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:39:25.513974 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:25.513950    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:25.515568 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:25.515542    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 18:39:25.515676 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:25.515592    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n" podUID="ec4eaec1-65e4-4507-b4fd-7938a9adcd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 18:39:26.514906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:26.514869    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 18:39:26.515306 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:26.514932    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n" podUID="ec4eaec1-65e4-4507-b4fd-7938a9adcd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 18:39:27.357730 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:27.357700    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:27.514712 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:27.514681    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 18:39:27.514877 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:27.514738    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n" podUID="ec4eaec1-65e4-4507-b4fd-7938a9adcd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 18:39:28.514963 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:28.514928    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 18:39:28.515354 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:28.514982    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n" podUID="ec4eaec1-65e4-4507-b4fd-7938a9adcd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 18:39:29.514164 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:29.514132    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 18:39:29.514346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:29.514196    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n" podUID="ec4eaec1-65e4-4507-b4fd-7938a9adcd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 18:39:30.439466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:30.439430    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"]
Oct 31 18:39:30.457132 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:30.457090    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:30.515243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:30.515207    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 18:39:30.515407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:30.515265    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n" podUID="ec4eaec1-65e4-4507-b4fd-7938a9adcd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 18:39:31.518925 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:31.518889    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:31.519336 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:31.519295    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:31.520043 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:31.520026    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:39:32.202895 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:32.202856    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:36.877460 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:36.877382    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:37.438519 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:37.438461    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:39:39.630661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.630631    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"]
Oct 31 18:39:39.634863 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.634844    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:39.693583 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.693550    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"]
Oct 31 18:39:39.740186 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.740154    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:39:39.740377 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.740313    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:39:39.746685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.746658    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gp8t\" (UniqueName: \"kubernetes.io/projected/705003bc-7509-4637-88c7-267931dfcc77-kube-api-access-4gp8t\") pod \"maas-api-revoke-sub-a990b93f-5n52h\" (UID: \"705003bc-7509-4637-88c7-267931dfcc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:39.746843 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.746708    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705003bc-7509-4637-88c7-267931dfcc77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a990b93f-5n52h\" (UID: \"705003bc-7509-4637-88c7-267931dfcc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:39.746944 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.746930    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705003bc-7509-4637-88c7-267931dfcc77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a990b93f-5n52h\" (UID: \"705003bc-7509-4637-88c7-267931dfcc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:39.848580 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.848549    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705003bc-7509-4637-88c7-267931dfcc77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a990b93f-5n52h\" (UID: \"705003bc-7509-4637-88c7-267931dfcc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:39.848774 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.848607    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gp8t\" (UniqueName: \"kubernetes.io/projected/705003bc-7509-4637-88c7-267931dfcc77-kube-api-access-4gp8t\") pod \"maas-api-revoke-sub-a990b93f-5n52h\" (UID: \"705003bc-7509-4637-88c7-267931dfcc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:39.848774 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.848647    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705003bc-7509-4637-88c7-267931dfcc77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a990b93f-5n52h\" (UID: \"705003bc-7509-4637-88c7-267931dfcc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:39.849368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.849347    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705003bc-7509-4637-88c7-267931dfcc77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a990b93f-5n52h\" (UID: \"705003bc-7509-4637-88c7-267931dfcc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:39.851027 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.851005    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705003bc-7509-4637-88c7-267931dfcc77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a990b93f-5n52h\" (UID: \"705003bc-7509-4637-88c7-267931dfcc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:39.906235 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.906165    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gp8t\" (UniqueName: \"kubernetes.io/projected/705003bc-7509-4637-88c7-267931dfcc77-kube-api-access-4gp8t\") pod \"maas-api-revoke-sub-a990b93f-5n52h\" (UID: \"705003bc-7509-4637-88c7-267931dfcc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:39.948445 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:39.948413    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:40.324135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:40.324110    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"]
Oct 31 18:39:40.324257 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:39:40.324194    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705003bc_7509_4637_88c7_267931dfcc77.slice/crio-485892acaf478fb3647e922f9f5a544a4099fe726a415d683e70bdeb9e8377db WatchSource:0}: Error finding container 485892acaf478fb3647e922f9f5a544a4099fe726a415d683e70bdeb9e8377db: Status 404 returned error can't find the container with id 485892acaf478fb3647e922f9f5a544a4099fe726a415d683e70bdeb9e8377db
Oct 31 18:39:40.750735 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:40.750698    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h" event={"ID":"705003bc-7509-4637-88c7-267931dfcc77","Type":"ContainerStarted","Data":"a4620d63ef542821ef96ab22c75cb538498cb71220105b82426a4e41ecd18fb2"}
Oct 31 18:39:40.751170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:40.750758    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h" event={"ID":"705003bc-7509-4637-88c7-267931dfcc77","Type":"ContainerStarted","Data":"485892acaf478fb3647e922f9f5a544a4099fe726a415d683e70bdeb9e8377db"}
Oct 31 18:39:40.866685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:40.866596    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h" podStartSLOduration=1.866581945 podStartE2EDuration="1.866581945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:39:40.866155906 +0000 UTC m=+1127.012605771" watchObservedRunningTime="2026-10-31 18:39:40.866581945 +0000 UTC m=+1127.013031810"
Oct 31 18:39:41.555409 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:41.555375    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:41.765502 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:41.765450    4484 generic.go:356] "Generic (PLEG): container finished" podID="705003bc-7509-4637-88c7-267931dfcc77" containerID="a4620d63ef542821ef96ab22c75cb538498cb71220105b82426a4e41ecd18fb2" exitCode=0
Oct 31 18:39:41.765917 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:41.765545    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h" event={"ID":"705003bc-7509-4637-88c7-267931dfcc77","Type":"ContainerDied","Data":"a4620d63ef542821ef96ab22c75cb538498cb71220105b82426a4e41ecd18fb2"}
Oct 31 18:39:42.910687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:42.910659    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:43.083564 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.083457    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705003bc-7509-4637-88c7-267931dfcc77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705003bc-7509-4637-88c7-267931dfcc77-maas-api-openshift-ca\") pod \"705003bc-7509-4637-88c7-267931dfcc77\" (UID: \"705003bc-7509-4637-88c7-267931dfcc77\") "
Oct 31 18:39:43.083711 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.083581    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705003bc-7509-4637-88c7-267931dfcc77-kube-api-access-4gp8t\" (UniqueName: \"kubernetes.io/projected/705003bc-7509-4637-88c7-267931dfcc77-kube-api-access-4gp8t\") pod \"705003bc-7509-4637-88c7-267931dfcc77\" (UID: \"705003bc-7509-4637-88c7-267931dfcc77\") "
Oct 31 18:39:43.083711 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.083625    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/705003bc-7509-4637-88c7-267931dfcc77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705003bc-7509-4637-88c7-267931dfcc77-maas-api-xks-ca\") pod \"705003bc-7509-4637-88c7-267931dfcc77\" (UID: \"705003bc-7509-4637-88c7-267931dfcc77\") "
Oct 31 18:39:43.083816 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.083793    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705003bc-7509-4637-88c7-267931dfcc77-maas-api-openshift-ca" pod "705003bc-7509-4637-88c7-267931dfcc77" (UID: "705003bc-7509-4637-88c7-267931dfcc77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:39:43.084126 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.084107    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705003bc-7509-4637-88c7-267931dfcc77-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:43.085891 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.085865    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705003bc-7509-4637-88c7-267931dfcc77-maas-api-xks-ca" pod "705003bc-7509-4637-88c7-267931dfcc77" (UID: "705003bc-7509-4637-88c7-267931dfcc77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:39:43.085989 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.085886    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705003bc-7509-4637-88c7-267931dfcc77-kube-api-access-4gp8t" pod "705003bc-7509-4637-88c7-267931dfcc77" (UID: "705003bc-7509-4637-88c7-267931dfcc77"). InnerVolumeSpecName "kube-api-access-4gp8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:39:43.185489 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.185449    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gp8t\" (UniqueName: \"kubernetes.io/projected/705003bc-7509-4637-88c7-267931dfcc77-kube-api-access-4gp8t\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:43.185489 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.185488    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705003bc-7509-4637-88c7-267931dfcc77-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:43.778823 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.778751    4484 generic.go:356] "Generic (PLEG): container finished" podID="4ab9d0e3-e0df-49d0-b281-d91156fad243" containerID="19acab58e86d7784f0796933b14dafd4fee17f655a1f1b1e49ae9d1e86145016" exitCode=0
Oct 31 18:39:43.779028 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.778877    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bcvft-hzf9d" event={"ID":"4ab9d0e3-e0df-49d0-b281-d91156fad243","Type":"ContainerDied","Data":"19acab58e86d7784f0796933b14dafd4fee17f655a1f1b1e49ae9d1e86145016"}
Oct 31 18:39:43.780362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.780342    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"
Oct 31 18:39:43.780362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.780349    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h" event={"ID":"705003bc-7509-4637-88c7-267931dfcc77","Type":"ContainerDied","Data":"485892acaf478fb3647e922f9f5a544a4099fe726a415d683e70bdeb9e8377db"}
Oct 31 18:39:43.780592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:43.780382    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="485892acaf478fb3647e922f9f5a544a4099fe726a415d683e70bdeb9e8377db"
Oct 31 18:39:44.931806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:44.931784    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:39:45.106085 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.106005    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-service-ca-bundle\") pod \"4ab9d0e3-e0df-49d0-b281-d91156fad243\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") "
Oct 31 18:39:45.106085 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.106054    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ab9d0e3-e0df-49d0-b281-d91156fad243-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ab9d0e3-e0df-49d0-b281-d91156fad243-archives-path\") pod \"4ab9d0e3-e0df-49d0-b281-d91156fad243\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") "
Oct 31 18:39:45.106298 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.106091    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-trusted-ca-bundle\") pod \"4ab9d0e3-e0df-49d0-b281-d91156fad243\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") "
Oct 31 18:39:45.106298 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.106173    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab9d0e3-e0df-49d0-b281-d91156fad243-kube-api-access-dlk5m\" (UniqueName: \"kubernetes.io/projected/4ab9d0e3-e0df-49d0-b281-d91156fad243-kube-api-access-dlk5m\") pod \"4ab9d0e3-e0df-49d0-b281-d91156fad243\" (UID: \"4ab9d0e3-e0df-49d0-b281-d91156fad243\") "
Oct 31 18:39:45.106437 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.106415    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ab9d0e3-e0df-49d0-b281-d91156fad243-archives-path" pod "4ab9d0e3-e0df-49d0-b281-d91156fad243" (UID: "4ab9d0e3-e0df-49d0-b281-d91156fad243"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:39:45.106524 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.106460    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-service-ca-bundle" pod "4ab9d0e3-e0df-49d0-b281-d91156fad243" (UID: "4ab9d0e3-e0df-49d0-b281-d91156fad243"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:39:45.106601 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.106576    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-trusted-ca-bundle" pod "4ab9d0e3-e0df-49d0-b281-d91156fad243" (UID: "4ab9d0e3-e0df-49d0-b281-d91156fad243"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:39:45.106930 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.106908    4484 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-service-ca-bundle\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:45.107067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.106935    4484 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4ab9d0e3-e0df-49d0-b281-d91156fad243-archives-path\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:45.107067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.106950    4484 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ab9d0e3-e0df-49d0-b281-d91156fad243-trusted-ca-bundle\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:45.108601 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.108581    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab9d0e3-e0df-49d0-b281-d91156fad243-kube-api-access-dlk5m" pod "4ab9d0e3-e0df-49d0-b281-d91156fad243" (UID: "4ab9d0e3-e0df-49d0-b281-d91156fad243"). InnerVolumeSpecName "kube-api-access-dlk5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:39:45.208010 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.207980    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlk5m\" (UniqueName: \"kubernetes.io/projected/4ab9d0e3-e0df-49d0-b281-d91156fad243-kube-api-access-dlk5m\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:45.459890 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.459790    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm" podUID="eec4901f-617d-482d-a4df-143749c1eeb7" containerName="main" containerID="cri-o://aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a" gracePeriod=45
Oct 31 18:39:45.730323 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.730301    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:39:45.795647 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.795618    4484 generic.go:356] "Generic (PLEG): container finished" podID="eec4901f-617d-482d-a4df-143749c1eeb7" containerID="aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a" exitCode=0
Oct 31 18:39:45.795785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.795671    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"
Oct 31 18:39:45.795785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.795704    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm" event={"ID":"eec4901f-617d-482d-a4df-143749c1eeb7","Type":"ContainerDied","Data":"aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a"}
Oct 31 18:39:45.795785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.795745    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm" event={"ID":"eec4901f-617d-482d-a4df-143749c1eeb7","Type":"ContainerDied","Data":"777dabe1ec5ff781209fbd0655bc410d40efd8a7a6fce7fedd8f7ad7b78d7783"}
Oct 31 18:39:45.795785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.795764    4484 scope.go:122] "RemoveContainer" containerID="aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a"
Oct 31 18:39:45.797044 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.797027    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bcvft-hzf9d" event={"ID":"4ab9d0e3-e0df-49d0-b281-d91156fad243","Type":"ContainerDied","Data":"078604ab5ef87ad3b27dfe2a925fefccec900b4ae362f782c9519ce426512e41"}
Oct 31 18:39:45.797122 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.797077    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bcvft-hzf9d"
Oct 31 18:39:45.797170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.797081    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="078604ab5ef87ad3b27dfe2a925fefccec900b4ae362f782c9519ce426512e41"
Oct 31 18:39:45.805195 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.805179    4484 scope.go:122] "RemoveContainer" containerID="aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a"
Oct 31 18:39:45.805458 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:39:45.805441    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a\": container with ID starting with aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a not found: ID does not exist" containerID="aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a"
Oct 31 18:39:45.805530 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.805465    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a"} err="failed to get container status \"aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a\": rpc error: code = NotFound desc = could not find container \"aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a\": container with ID starting with aacf750714590ca1b9473be410b3aee9ee7337bba1a8dc4036833bf7ec452d0a not found: ID does not exist"
Oct 31 18:39:45.916181 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.916155    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-dshm\") pod \"eec4901f-617d-482d-a4df-143749c1eeb7\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") "
Oct 31 18:39:45.916181 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.916184    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-tmp-dir\") pod \"eec4901f-617d-482d-a4df-143749c1eeb7\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") "
Oct 31 18:39:45.916396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.916358    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-home\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-home\") pod \"eec4901f-617d-482d-a4df-143749c1eeb7\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") "
Oct 31 18:39:45.916396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.916387    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-model-cache\") pod \"eec4901f-617d-482d-a4df-143749c1eeb7\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") "
Oct 31 18:39:45.916538 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.916465    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec4901f-617d-482d-a4df-143749c1eeb7-kube-api-access-dzl9g\" (UniqueName: \"kubernetes.io/projected/eec4901f-617d-482d-a4df-143749c1eeb7-kube-api-access-dzl9g\") pod \"eec4901f-617d-482d-a4df-143749c1eeb7\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") "
Oct 31 18:39:45.916595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.916538    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-tmp-dir" pod "eec4901f-617d-482d-a4df-143749c1eeb7" (UID: "eec4901f-617d-482d-a4df-143749c1eeb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:39:45.916644 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.916605    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eec4901f-617d-482d-a4df-143749c1eeb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/eec4901f-617d-482d-a4df-143749c1eeb7-tls-certs\") pod \"eec4901f-617d-482d-a4df-143749c1eeb7\" (UID: \"eec4901f-617d-482d-a4df-143749c1eeb7\") "
Oct 31 18:39:45.916644 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.916630    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-home" pod "eec4901f-617d-482d-a4df-143749c1eeb7" (UID: "eec4901f-617d-482d-a4df-143749c1eeb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:39:45.916725 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.916705    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-model-cache" pod "eec4901f-617d-482d-a4df-143749c1eeb7" (UID: "eec4901f-617d-482d-a4df-143749c1eeb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:39:45.917043 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.917023    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:45.917043 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.917046    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:45.917223 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.917083    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:45.918539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.918516    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-dshm" pod "eec4901f-617d-482d-a4df-143749c1eeb7" (UID: "eec4901f-617d-482d-a4df-143749c1eeb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:39:45.918751 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.918731    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec4901f-617d-482d-a4df-143749c1eeb7-tls-certs" pod "eec4901f-617d-482d-a4df-143749c1eeb7" (UID: "eec4901f-617d-482d-a4df-143749c1eeb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:39:45.918927 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:45.918903    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec4901f-617d-482d-a4df-143749c1eeb7-kube-api-access-dzl9g" pod "eec4901f-617d-482d-a4df-143749c1eeb7" (UID: "eec4901f-617d-482d-a4df-143749c1eeb7"). InnerVolumeSpecName "kube-api-access-dzl9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:39:46.017873 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:46.017845    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzl9g\" (UniqueName: \"kubernetes.io/projected/eec4901f-617d-482d-a4df-143749c1eeb7-kube-api-access-dzl9g\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:46.017873 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:46.017870    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec4901f-617d-482d-a4df-143749c1eeb7-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:46.018210 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:46.017879    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec4901f-617d-482d-a4df-143749c1eeb7-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:46.212344 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:46.212317    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"]
Oct 31 18:39:46.214741 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:46.214722    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-88085e/e2e-conflict-b-1a3e1910-kserve-9cb644f94-9w6bm"]
Oct 31 18:39:46.252724 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:46.252556    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:46.443837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:46.443794    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:46.763271 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:46.763240    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:46.923247 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:46.923205    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:47.092603 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:47.092519    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:47.993765 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:47.993738    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec4901f-617d-482d-a4df-143749c1eeb7" path="/var/lib/kubelet/pods/eec4901f-617d-482d-a4df-143749c1eeb7/volumes"
Oct 31 18:39:50.959645 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:50.959614    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:51.157203 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:51.157172    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:52.847458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.847426    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"]
Oct 31 18:39:52.847971 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.847957    4484 cpu_manager.go:414] "containerMap: removing container" podUID="705003bc-7509-4637-88c7-267931dfcc77" containerName="revoke-keys"
Oct 31 18:39:52.848017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.847972    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705003bc-7509-4637-88c7-267931dfcc77" containerName="revoke-keys"
Oct 31 18:39:52.848017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.848004    4484 cpu_manager.go:414] "containerMap: removing container" podUID="4ab9d0e3-e0df-49d0-b281-d91156fad243" containerName="insights-gathering"
Oct 31 18:39:52.848017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.848009    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab9d0e3-e0df-49d0-b281-d91156fad243" containerName="insights-gathering"
Oct 31 18:39:52.848017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.848017    4484 cpu_manager.go:414] "containerMap: removing container" podUID="eec4901f-617d-482d-a4df-143749c1eeb7" containerName="main"
Oct 31 18:39:52.848139 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.848021    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec4901f-617d-482d-a4df-143749c1eeb7" containerName="main"
Oct 31 18:39:52.848139 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.848102    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab9d0e3-e0df-49d0-b281-d91156fad243" containerName="insights-gathering"
Oct 31 18:39:52.848139 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.848108    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec4901f-617d-482d-a4df-143749c1eeb7" containerName="main"
Oct 31 18:39:52.848139 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.848116    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="705003bc-7509-4637-88c7-267931dfcc77" containerName="revoke-keys"
Oct 31 18:39:52.852689 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.852672    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:52.887890 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.887867    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4387450c-ld8xt\" (UID: \"e096445e-efea-4b11-b690-f9fcbf00b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:52.888027 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.887894    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvbf4\" (UniqueName: \"kubernetes.io/projected/e096445e-efea-4b11-b690-f9fcbf00b109-kube-api-access-wvbf4\") pod \"maas-api-revoke-sub-4387450c-ld8xt\" (UID: \"e096445e-efea-4b11-b690-f9fcbf00b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:52.888027 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.887924    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4387450c-ld8xt\" (UID: \"e096445e-efea-4b11-b690-f9fcbf00b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:52.906558 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.906535    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"]
Oct 31 18:39:52.953876 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.953855    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:39:52.954066 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.954051    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:39:52.988962 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.988932    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4387450c-ld8xt\" (UID: \"e096445e-efea-4b11-b690-f9fcbf00b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:52.989109 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.988969    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvbf4\" (UniqueName: \"kubernetes.io/projected/e096445e-efea-4b11-b690-f9fcbf00b109-kube-api-access-wvbf4\") pod \"maas-api-revoke-sub-4387450c-ld8xt\" (UID: \"e096445e-efea-4b11-b690-f9fcbf00b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:52.989109 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.989003    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4387450c-ld8xt\" (UID: \"e096445e-efea-4b11-b690-f9fcbf00b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:52.989723 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.989700    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4387450c-ld8xt\" (UID: \"e096445e-efea-4b11-b690-f9fcbf00b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:52.991357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:52.991336    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4387450c-ld8xt\" (UID: \"e096445e-efea-4b11-b690-f9fcbf00b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:53.046367 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:53.046339    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvbf4\" (UniqueName: \"kubernetes.io/projected/e096445e-efea-4b11-b690-f9fcbf00b109-kube-api-access-wvbf4\") pod \"maas-api-revoke-sub-4387450c-ld8xt\" (UID: \"e096445e-efea-4b11-b690-f9fcbf00b109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:53.165410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:53.165340    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:53.539923 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:53.539901    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"]
Oct 31 18:39:53.540504 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:39:53.540458    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode096445e_efea_4b11_b690_f9fcbf00b109.slice/crio-950057950f2aba2a995de2eaf9da7f7c759f918a660cb8128f5f9690e4a2f0db WatchSource:0}: Error finding container 950057950f2aba2a995de2eaf9da7f7c759f918a660cb8128f5f9690e4a2f0db: Status 404 returned error can't find the container with id 950057950f2aba2a995de2eaf9da7f7c759f918a660cb8128f5f9690e4a2f0db
Oct 31 18:39:53.841862 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:53.841781    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt" event={"ID":"e096445e-efea-4b11-b690-f9fcbf00b109","Type":"ContainerStarted","Data":"950057950f2aba2a995de2eaf9da7f7c759f918a660cb8128f5f9690e4a2f0db"}
Oct 31 18:39:54.848593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:54.848554    4484 generic.go:356] "Generic (PLEG): container finished" podID="e096445e-efea-4b11-b690-f9fcbf00b109" containerID="ffcc25b8b511d6f6e9b1c5fda9ff349568f3ac085a8b558d98e490a00bed9489" exitCode=0
Oct 31 18:39:54.848987 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:54.848613    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt" event={"ID":"e096445e-efea-4b11-b690-f9fcbf00b109","Type":"ContainerDied","Data":"ffcc25b8b511d6f6e9b1c5fda9ff349568f3ac085a8b558d98e490a00bed9489"}
Oct 31 18:39:55.467323 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:39:55.467284    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:39:55.766946 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:55.766916    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:39:55.992510 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:55.992460    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:56.123871 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:56.123781    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-openshift-ca\") pod \"e096445e-efea-4b11-b690-f9fcbf00b109\" (UID: \"e096445e-efea-4b11-b690-f9fcbf00b109\") "
Oct 31 18:39:56.124030 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:56.123964    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-xks-ca\") pod \"e096445e-efea-4b11-b690-f9fcbf00b109\" (UID: \"e096445e-efea-4b11-b690-f9fcbf00b109\") "
Oct 31 18:39:56.124030 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:56.123995    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e096445e-efea-4b11-b690-f9fcbf00b109-kube-api-access-wvbf4\" (UniqueName: \"kubernetes.io/projected/e096445e-efea-4b11-b690-f9fcbf00b109-kube-api-access-wvbf4\") pod \"e096445e-efea-4b11-b690-f9fcbf00b109\" (UID: \"e096445e-efea-4b11-b690-f9fcbf00b109\") "
Oct 31 18:39:56.124180 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:56.124154    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-openshift-ca" pod "e096445e-efea-4b11-b690-f9fcbf00b109" (UID: "e096445e-efea-4b11-b690-f9fcbf00b109"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:39:56.124395 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:56.124376    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:56.126523 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:56.126500    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e096445e-efea-4b11-b690-f9fcbf00b109-kube-api-access-wvbf4" pod "e096445e-efea-4b11-b690-f9fcbf00b109" (UID: "e096445e-efea-4b11-b690-f9fcbf00b109"). InnerVolumeSpecName "kube-api-access-wvbf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:39:56.132710 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:56.132685    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-xks-ca" pod "e096445e-efea-4b11-b690-f9fcbf00b109" (UID: "e096445e-efea-4b11-b690-f9fcbf00b109"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:39:56.225965 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:56.225939    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e096445e-efea-4b11-b690-f9fcbf00b109-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:56.225965 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:56.225963    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvbf4\" (UniqueName: \"kubernetes.io/projected/e096445e-efea-4b11-b690-f9fcbf00b109-kube-api-access-wvbf4\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:39:56.861321 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:56.861276    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt" event={"ID":"e096445e-efea-4b11-b690-f9fcbf00b109","Type":"ContainerDied","Data":"950057950f2aba2a995de2eaf9da7f7c759f918a660cb8128f5f9690e4a2f0db"}
Oct 31 18:39:56.861321 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:56.861300    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"
Oct 31 18:39:56.861596 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:39:56.861330    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="950057950f2aba2a995de2eaf9da7f7c759f918a660cb8128f5f9690e4a2f0db"
Oct 31 18:40:00.462649 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:00.462614    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:05.153541 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:05.153505    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:08.792863 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.792831    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"]
Oct 31 18:40:08.793384 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.793371    4484 cpu_manager.go:414] "containerMap: removing container" podUID="e096445e-efea-4b11-b690-f9fcbf00b109" containerName="revoke-keys"
Oct 31 18:40:08.793384 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.793385    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e096445e-efea-4b11-b690-f9fcbf00b109" containerName="revoke-keys"
Oct 31 18:40:08.793560 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.793521    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e096445e-efea-4b11-b690-f9fcbf00b109" containerName="revoke-keys"
Oct 31 18:40:08.797164 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.797141    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:08.852422 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.852392    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"]
Oct 31 18:40:08.895923 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.895898    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872c5b28-rhbmz\" (UID: \"424b4a98-595d-4edb-ae17-a4915eb278c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:08.896073 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.895962    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdglp\" (UniqueName: \"kubernetes.io/projected/424b4a98-595d-4edb-ae17-a4915eb278c2-kube-api-access-cdglp\") pod \"maas-api-revoke-sub-872c5b28-rhbmz\" (UID: \"424b4a98-595d-4edb-ae17-a4915eb278c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:08.896127 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.896109    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872c5b28-rhbmz\" (UID: \"424b4a98-595d-4edb-ae17-a4915eb278c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:08.899486 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.899452    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:40:08.899600 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.899527    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:40:08.997342 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.997310    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872c5b28-rhbmz\" (UID: \"424b4a98-595d-4edb-ae17-a4915eb278c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:08.997525 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.997381    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdglp\" (UniqueName: \"kubernetes.io/projected/424b4a98-595d-4edb-ae17-a4915eb278c2-kube-api-access-cdglp\") pod \"maas-api-revoke-sub-872c5b28-rhbmz\" (UID: \"424b4a98-595d-4edb-ae17-a4915eb278c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:08.997525 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.997425    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872c5b28-rhbmz\" (UID: \"424b4a98-595d-4edb-ae17-a4915eb278c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:08.998008 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.997983    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872c5b28-rhbmz\" (UID: \"424b4a98-595d-4edb-ae17-a4915eb278c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:08.999849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:08.999829    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872c5b28-rhbmz\" (UID: \"424b4a98-595d-4edb-ae17-a4915eb278c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:09.054861 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:09.054797    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdglp\" (UniqueName: \"kubernetes.io/projected/424b4a98-595d-4edb-ae17-a4915eb278c2-kube-api-access-cdglp\") pod \"maas-api-revoke-sub-872c5b28-rhbmz\" (UID: \"424b4a98-595d-4edb-ae17-a4915eb278c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:09.110833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:09.110800    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:09.494995 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:40:09.494955    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424b4a98_595d_4edb_ae17_a4915eb278c2.slice/crio-4f82906fa2049ebd7366d3154371340c28f5e4f95c0d914c704f6904627d4e4c WatchSource:0}: Error finding container 4f82906fa2049ebd7366d3154371340c28f5e4f95c0d914c704f6904627d4e4c: Status 404 returned error can't find the container with id 4f82906fa2049ebd7366d3154371340c28f5e4f95c0d914c704f6904627d4e4c
Oct 31 18:40:09.495132 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:09.495084    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"]
Oct 31 18:40:09.791525 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:09.791494    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:09.994543 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:09.994506    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz" event={"ID":"424b4a98-595d-4edb-ae17-a4915eb278c2","Type":"ContainerStarted","Data":"95415b36551f7c310225222dedcc26c2200f62343b161fc1dc09473afc546ee7"}
Oct 31 18:40:09.994939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:09.994565    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz" event={"ID":"424b4a98-595d-4edb-ae17-a4915eb278c2","Type":"ContainerStarted","Data":"4f82906fa2049ebd7366d3154371340c28f5e4f95c0d914c704f6904627d4e4c"}
Oct 31 18:40:10.110772 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:10.110681    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz" podStartSLOduration=2.110667947 podStartE2EDuration="2.110667947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:10.108560055 +0000 UTC m=+1156.255009919" watchObservedRunningTime="2026-10-31 18:40:10.110667947 +0000 UTC m=+1156.257117811"
Oct 31 18:40:11.001699 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:11.001670    4484 generic.go:356] "Generic (PLEG): container finished" podID="424b4a98-595d-4edb-ae17-a4915eb278c2" containerID="95415b36551f7c310225222dedcc26c2200f62343b161fc1dc09473afc546ee7" exitCode=0
Oct 31 18:40:11.002157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:11.001769    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz" event={"ID":"424b4a98-595d-4edb-ae17-a4915eb278c2","Type":"ContainerDied","Data":"95415b36551f7c310225222dedcc26c2200f62343b161fc1dc09473afc546ee7"}
Oct 31 18:40:12.143926 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:12.143904    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:12.232194 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:12.232162    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424b4a98-595d-4edb-ae17-a4915eb278c2-kube-api-access-cdglp\" (UniqueName: \"kubernetes.io/projected/424b4a98-595d-4edb-ae17-a4915eb278c2-kube-api-access-cdglp\") pod \"424b4a98-595d-4edb-ae17-a4915eb278c2\" (UID: \"424b4a98-595d-4edb-ae17-a4915eb278c2\") "
Oct 31 18:40:12.232360 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:12.232237    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-openshift-ca\") pod \"424b4a98-595d-4edb-ae17-a4915eb278c2\" (UID: \"424b4a98-595d-4edb-ae17-a4915eb278c2\") "
Oct 31 18:40:12.232360 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:12.232317    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-xks-ca\") pod \"424b4a98-595d-4edb-ae17-a4915eb278c2\" (UID: \"424b4a98-595d-4edb-ae17-a4915eb278c2\") "
Oct 31 18:40:12.232618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:12.232596    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-openshift-ca" pod "424b4a98-595d-4edb-ae17-a4915eb278c2" (UID: "424b4a98-595d-4edb-ae17-a4915eb278c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:40:12.232826 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:12.232809    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:12.234391 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:12.234372    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-xks-ca" pod "424b4a98-595d-4edb-ae17-a4915eb278c2" (UID: "424b4a98-595d-4edb-ae17-a4915eb278c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:40:12.234535 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:12.234516    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424b4a98-595d-4edb-ae17-a4915eb278c2-kube-api-access-cdglp" pod "424b4a98-595d-4edb-ae17-a4915eb278c2" (UID: "424b4a98-595d-4edb-ae17-a4915eb278c2"). InnerVolumeSpecName "kube-api-access-cdglp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:40:12.334313 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:12.334226    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424b4a98-595d-4edb-ae17-a4915eb278c2-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:12.334313 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:12.334257    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdglp\" (UniqueName: \"kubernetes.io/projected/424b4a98-595d-4edb-ae17-a4915eb278c2-kube-api-access-cdglp\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:13.012871 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:13.012838    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz" event={"ID":"424b4a98-595d-4edb-ae17-a4915eb278c2","Type":"ContainerDied","Data":"4f82906fa2049ebd7366d3154371340c28f5e4f95c0d914c704f6904627d4e4c"}
Oct 31 18:40:13.013037 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:13.012874    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"
Oct 31 18:40:13.013037 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:13.012884    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f82906fa2049ebd7366d3154371340c28f5e4f95c0d914c704f6904627d4e4c"
Oct 31 18:40:14.355992 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:14.355958    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:40:14.444890 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:14.444861    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:19.173065 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:19.173031    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:23.650938 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.650907    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"]
Oct 31 18:40:23.651506 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.651492    4484 cpu_manager.go:414] "containerMap: removing container" podUID="424b4a98-595d-4edb-ae17-a4915eb278c2" containerName="revoke-keys"
Oct 31 18:40:23.651558 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.651507    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424b4a98-595d-4edb-ae17-a4915eb278c2" containerName="revoke-keys"
Oct 31 18:40:23.651592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.651586    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="424b4a98-595d-4edb-ae17-a4915eb278c2" containerName="revoke-keys"
Oct 31 18:40:23.655125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.655110    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.690305 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.690261    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.690456 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.690317    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.690456 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.690379    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.690456 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.690419    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.690677 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.690457    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.690732 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.690703    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.690786 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.690742    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.690883 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.690861    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j58r\" (UniqueName: \"kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-kube-api-access-8j58r\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.690944 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.690897    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.755723 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.755696    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"]
Oct 31 18:40:23.791710 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.791670    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.791901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.791716    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.791901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.791755    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j58r\" (UniqueName: \"kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-kube-api-access-8j58r\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.791901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.791781    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.791901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.791840    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.791901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.791882    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.792242 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.791947    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.792242 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.791997    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.792242 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.792041    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.792402 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.792378    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.792563 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.792542    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.792746 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.792637    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.792962 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.792933    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.793109 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.793083    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.794826 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.794807    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.794942 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.794877    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.848976 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.848953    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.849092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.849074    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j58r\" (UniqueName: \"kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-kube-api-access-8j58r\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:23.967609 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:23.967532    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:24.013733 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.013697    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:24.180428 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.180401    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"]
Oct 31 18:40:24.185299 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.185283    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.242246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.242220    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-da6ea7\"/\"distinct-2-w1-da6ea7-kserve-self-signed-certs\""
Oct 31 18:40:24.242404 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.242262    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-da6ea7\"/\"kube-root-ca.crt\""
Oct 31 18:40:24.243085 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.243061    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"]
Oct 31 18:40:24.258143 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.258123    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-da6ea7\"/\"openshift-service-ca.crt\""
Oct 31 18:40:24.258143 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.258123    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-da6ea7\"/\"default-dockercfg-dchgm\""
Oct 31 18:40:24.289875 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.289846    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:24.296974 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.296951    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tmp-dir\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.297084 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.297013    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tls-certs\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.297128 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.297105    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-home\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.297208 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.297185    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-model-cache\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.297251 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.297234    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-dshm\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.297315 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.297291    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqxtl\" (UniqueName: \"kubernetes.io/projected/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-kube-api-access-gqxtl\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.394395 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.394358    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"]
Oct 31 18:40:24.394919 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:40:24.394895    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ee432c_0300_43c5_8cb5_c4606ed6e23b.slice/crio-853914bb886370f40da4c141c5311fe489ddf63d263507772fc0f5633921e42e WatchSource:0}: Error finding container 853914bb886370f40da4c141c5311fe489ddf63d263507772fc0f5633921e42e: Status 404 returned error can't find the container with id 853914bb886370f40da4c141c5311fe489ddf63d263507772fc0f5633921e42e
Oct 31 18:40:24.397211 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.397180    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:40:24.397280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.397245    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:40:24.397280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.397274    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:40:24.398014 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.397996    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tls-certs\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.398122 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.398028    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-home\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.398122 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.398063    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-model-cache\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.398122 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.398103    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-dshm\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.398278 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.398124    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqxtl\" (UniqueName: \"kubernetes.io/projected/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-kube-api-access-gqxtl\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.398278 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.398216    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tmp-dir\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.398436 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.398417    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-home\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.398516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.398456    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-model-cache\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.398644 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.398624    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tmp-dir\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.400419 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.400397    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-dshm\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.400661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.400644    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tls-certs\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.455233 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.455206    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqxtl\" (UniqueName: \"kubernetes.io/projected/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-kube-api-access-gqxtl\") pod \"distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") " pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.497686 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.497602    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:24.877679 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:40:24.877648    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2ff42d7_b4f9_4b25_b2b7_1e429f280aec.slice/crio-008804d04750bb650d6cd686d9245feb53806339e0ff7c877c414009c4f5a8fa WatchSource:0}: Error finding container 008804d04750bb650d6cd686d9245feb53806339e0ff7c877c414009c4f5a8fa: Status 404 returned error can't find the container with id 008804d04750bb650d6cd686d9245feb53806339e0ff7c877c414009c4f5a8fa
Oct 31 18:40:24.883878 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:24.883853    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"]
Oct 31 18:40:25.101374 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.101334    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84" event={"ID":"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec","Type":"ContainerStarted","Data":"af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90"}
Oct 31 18:40:25.101556 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.101387    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84" event={"ID":"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec","Type":"ContainerStarted","Data":"008804d04750bb650d6cd686d9245feb53806339e0ff7c877c414009c4f5a8fa"}
Oct 31 18:40:25.103067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.103041    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc" event={"ID":"76ee432c-0300-43c5-8cb5-c4606ed6e23b","Type":"ContainerStarted","Data":"7e7bd195b96f9075c74a8800e29bbff7fe639679834a95f8161b32173070dfdc"}
Oct 31 18:40:25.103193 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.103085    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc" event={"ID":"76ee432c-0300-43c5-8cb5-c4606ed6e23b","Type":"ContainerStarted","Data":"853914bb886370f40da4c141c5311fe489ddf63d263507772fc0f5633921e42e"}
Oct 31 18:40:25.219445 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.219352    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84" podStartSLOduration=1.219338358 podStartE2EDuration="1.219338358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:25.216739573 +0000 UTC m=+1171.363189436" watchObservedRunningTime="2026-10-31 18:40:25.219338358 +0000 UTC m=+1171.365788221"
Oct 31 18:40:25.352023 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.351970    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc" podStartSLOduration=2.351953178 podStartE2EDuration="2.351953178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:25.34867457 +0000 UTC m=+1171.495124435" watchObservedRunningTime="2026-10-31 18:40:25.351953178 +0000 UTC m=+1171.498403041"
Oct 31 18:40:25.491205 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:40:25.491160    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:40:25.498385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.498361    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:25.635639 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.635610    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"]
Oct 31 18:40:25.639687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.639662    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:25.692935 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.692898    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:40:25.693119 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.692980    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:40:25.697394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.697366    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"]
Oct 31 18:40:25.711971 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.711941    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-529f1630-8jtfs\" (UID: \"d694131f-187f-4bc5-a78c-db7dd7e754de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:25.712136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.712027    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-529f1630-8jtfs\" (UID: \"d694131f-187f-4bc5-a78c-db7dd7e754de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:25.712136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.712078    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj8f9\" (UniqueName: \"kubernetes.io/projected/d694131f-187f-4bc5-a78c-db7dd7e754de-kube-api-access-gj8f9\") pod \"maas-api-revoke-sub-529f1630-8jtfs\" (UID: \"d694131f-187f-4bc5-a78c-db7dd7e754de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:25.812816 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.812713    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-529f1630-8jtfs\" (UID: \"d694131f-187f-4bc5-a78c-db7dd7e754de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:25.812816 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.812780    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-529f1630-8jtfs\" (UID: \"d694131f-187f-4bc5-a78c-db7dd7e754de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:25.813050 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.812827    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj8f9\" (UniqueName: \"kubernetes.io/projected/d694131f-187f-4bc5-a78c-db7dd7e754de-kube-api-access-gj8f9\") pod \"maas-api-revoke-sub-529f1630-8jtfs\" (UID: \"d694131f-187f-4bc5-a78c-db7dd7e754de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:25.813426 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.813404    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-529f1630-8jtfs\" (UID: \"d694131f-187f-4bc5-a78c-db7dd7e754de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:25.815242 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.815218    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-529f1630-8jtfs\" (UID: \"d694131f-187f-4bc5-a78c-db7dd7e754de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:25.887415 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.887370    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj8f9\" (UniqueName: \"kubernetes.io/projected/d694131f-187f-4bc5-a78c-db7dd7e754de-kube-api-access-gj8f9\") pod \"maas-api-revoke-sub-529f1630-8jtfs\" (UID: \"d694131f-187f-4bc5-a78c-db7dd7e754de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:25.953410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.953379    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:25.991539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.983142    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:25.991539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.986818    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 18:40:25.991539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:25.986873    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc" podUID="76ee432c-0300-43c5-8cb5-c4606ed6e23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 18:40:26.356648 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:26.356619    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"]
Oct 31 18:40:26.356998 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:40:26.356974    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd694131f_187f_4bc5_a78c_db7dd7e754de.slice/crio-f072ce0cac7b45a4a72463ea3b2f7f6c2459f6f2eedc585cb34cbabecd6d5a9c WatchSource:0}: Error finding container f072ce0cac7b45a4a72463ea3b2f7f6c2459f6f2eedc585cb34cbabecd6d5a9c: Status 404 returned error can't find the container with id f072ce0cac7b45a4a72463ea3b2f7f6c2459f6f2eedc585cb34cbabecd6d5a9c
Oct 31 18:40:26.981555 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:26.976206    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 18:40:26.981555 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:26.976266    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc" podUID="76ee432c-0300-43c5-8cb5-c4606ed6e23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 18:40:27.120385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:27.120343    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs" event={"ID":"d694131f-187f-4bc5-a78c-db7dd7e754de","Type":"ContainerStarted","Data":"e832cd2b1ebfcf039df34944bd83322c7683a99fcb0fa9212025d9503ed4986b"}
Oct 31 18:40:27.120574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:27.120397    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs" event={"ID":"d694131f-187f-4bc5-a78c-db7dd7e754de","Type":"ContainerStarted","Data":"f072ce0cac7b45a4a72463ea3b2f7f6c2459f6f2eedc585cb34cbabecd6d5a9c"}
Oct 31 18:40:27.246373 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:27.246323    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs" podStartSLOduration=2.246310742 podStartE2EDuration="2.246310742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:27.243231825 +0000 UTC m=+1173.389681691" watchObservedRunningTime="2026-10-31 18:40:27.246310742 +0000 UTC m=+1173.392760605"
Oct 31 18:40:27.984554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:27.980907    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 18:40:27.984554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:27.980968    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc" podUID="76ee432c-0300-43c5-8cb5-c4606ed6e23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 18:40:28.127835 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:28.127805    4484 generic.go:356] "Generic (PLEG): container finished" podID="d694131f-187f-4bc5-a78c-db7dd7e754de" containerID="e832cd2b1ebfcf039df34944bd83322c7683a99fcb0fa9212025d9503ed4986b" exitCode=0
Oct 31 18:40:28.128008 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:28.127895    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs" event={"ID":"d694131f-187f-4bc5-a78c-db7dd7e754de","Type":"ContainerDied","Data":"e832cd2b1ebfcf039df34944bd83322c7683a99fcb0fa9212025d9503ed4986b"}
Oct 31 18:40:28.903817 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:28.903787    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:28.975501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:28.967903    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 18:40:28.975501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:28.967993    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc" podUID="76ee432c-0300-43c5-8cb5-c4606ed6e23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 18:40:29.345071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:29.345040    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:29.456702 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:29.456670    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-xks-ca\") pod \"d694131f-187f-4bc5-a78c-db7dd7e754de\" (UID: \"d694131f-187f-4bc5-a78c-db7dd7e754de\") "
Oct 31 18:40:29.456849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:29.456736    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d694131f-187f-4bc5-a78c-db7dd7e754de-kube-api-access-gj8f9\" (UniqueName: \"kubernetes.io/projected/d694131f-187f-4bc5-a78c-db7dd7e754de-kube-api-access-gj8f9\") pod \"d694131f-187f-4bc5-a78c-db7dd7e754de\" (UID: \"d694131f-187f-4bc5-a78c-db7dd7e754de\") "
Oct 31 18:40:29.456849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:29.456757    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-openshift-ca\") pod \"d694131f-187f-4bc5-a78c-db7dd7e754de\" (UID: \"d694131f-187f-4bc5-a78c-db7dd7e754de\") "
Oct 31 18:40:29.457166 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:29.457133    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-openshift-ca" pod "d694131f-187f-4bc5-a78c-db7dd7e754de" (UID: "d694131f-187f-4bc5-a78c-db7dd7e754de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:40:29.458973 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:29.458943    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d694131f-187f-4bc5-a78c-db7dd7e754de-kube-api-access-gj8f9" pod "d694131f-187f-4bc5-a78c-db7dd7e754de" (UID: "d694131f-187f-4bc5-a78c-db7dd7e754de"). InnerVolumeSpecName "kube-api-access-gj8f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:40:29.459059 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:29.458982    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-xks-ca" pod "d694131f-187f-4bc5-a78c-db7dd7e754de" (UID: "d694131f-187f-4bc5-a78c-db7dd7e754de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:40:29.558642 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:29.558610    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj8f9\" (UniqueName: \"kubernetes.io/projected/d694131f-187f-4bc5-a78c-db7dd7e754de-kube-api-access-gj8f9\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:29.558642 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:29.558637    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:29.558642 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:29.558648    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d694131f-187f-4bc5-a78c-db7dd7e754de-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:29.973030 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:29.972995    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 18:40:29.973300 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:29.973270    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc" podUID="76ee432c-0300-43c5-8cb5-c4606ed6e23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 18:40:30.140935 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:30.140892    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs" event={"ID":"d694131f-187f-4bc5-a78c-db7dd7e754de","Type":"ContainerDied","Data":"f072ce0cac7b45a4a72463ea3b2f7f6c2459f6f2eedc585cb34cbabecd6d5a9c"}
Oct 31 18:40:30.140935 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:30.140941    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f072ce0cac7b45a4a72463ea3b2f7f6c2459f6f2eedc585cb34cbabecd6d5a9c"
Oct 31 18:40:30.141198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:30.140993    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"
Oct 31 18:40:30.972499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:30.971864    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 18:40:30.972499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:30.971927    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc" podUID="76ee432c-0300-43c5-8cb5-c4606ed6e23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 18:40:32.912973 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:32.912945    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:32.913399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:32.913268    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:32.914163 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:32.914146    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:40:33.952429 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:33.952395    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:34.498346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:34.498317    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:34.510993 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:34.510971    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:35.185788 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.185760    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 18:40:35.762071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.762039    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"]
Oct 31 18:40:35.762616 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.762600    4484 cpu_manager.go:414] "containerMap: removing container" podUID="d694131f-187f-4bc5-a78c-db7dd7e754de" containerName="revoke-keys"
Oct 31 18:40:35.762688 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.762616    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d694131f-187f-4bc5-a78c-db7dd7e754de" containerName="revoke-keys"
Oct 31 18:40:35.762732 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.762723    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d694131f-187f-4bc5-a78c-db7dd7e754de" containerName="revoke-keys"
Oct 31 18:40:35.766366 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.766348    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.866324 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.866293    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czxqv\" (UniqueName: \"kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-kube-api-access-czxqv\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.866533 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.866330    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.866533 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.866417    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.866533 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.866446    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.866533 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.866509    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.866702 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.866595    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.866702 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.866642    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.866702 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.866690    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.866812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.866712    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.873896 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.873868    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"]
Oct 31 18:40:35.968367 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.968337    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.968367 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.968373    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.968618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.968414    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.968618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.968438    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.968618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.968463    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.968618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.968547    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.968618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.968579    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.968861 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.968667    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czxqv\" (UniqueName: \"kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-kube-api-access-czxqv\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.968861 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.968697    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.969001 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.968974    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.969072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.969026    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.969157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.969137    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.969276 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.969169    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.969276 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.969201    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.971578 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.971554    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:35.971875 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:35.971856    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:36.026072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.026005    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:36.026227 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.026208    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czxqv\" (UniqueName: \"kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-kube-api-access-czxqv\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:36.080174 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.080150    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:36.371780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.371706    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"]
Oct 31 18:40:36.377006 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.376984    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:36.429836 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.429815    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:40:36.430059 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.429928    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:40:36.431580 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.431558    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"]
Oct 31 18:40:36.474013 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.473984    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c36c08b-8r5hc\" (UID: \"c422eb47-48de-4d6c-bc9d-812797f4cb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:36.474166 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.474048    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c36c08b-8r5hc\" (UID: \"c422eb47-48de-4d6c-bc9d-812797f4cb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:36.474244 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.474216    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2gn\" (UniqueName: \"kubernetes.io/projected/c422eb47-48de-4d6c-bc9d-812797f4cb26-kube-api-access-2h2gn\") pod \"maas-api-revoke-sub-1c36c08b-8r5hc\" (UID: \"c422eb47-48de-4d6c-bc9d-812797f4cb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:36.507520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.507495    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"]
Oct 31 18:40:36.508378 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:40:36.508351    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd5dbc4_d25a_4018_866e_2161b2378eaa.slice/crio-895f6ff07e8e84390e4845c6cb045746869a8ed02430d7d97f886ae8ac91217a WatchSource:0}: Error finding container 895f6ff07e8e84390e4845c6cb045746869a8ed02430d7d97f886ae8ac91217a: Status 404 returned error can't find the container with id 895f6ff07e8e84390e4845c6cb045746869a8ed02430d7d97f886ae8ac91217a
Oct 31 18:40:36.510784 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.510755    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:40:36.510848 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.510819    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:40:36.510880 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.510852    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:40:36.575723 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.575691    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c36c08b-8r5hc\" (UID: \"c422eb47-48de-4d6c-bc9d-812797f4cb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:36.575823 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.575783    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2gn\" (UniqueName: \"kubernetes.io/projected/c422eb47-48de-4d6c-bc9d-812797f4cb26-kube-api-access-2h2gn\") pod \"maas-api-revoke-sub-1c36c08b-8r5hc\" (UID: \"c422eb47-48de-4d6c-bc9d-812797f4cb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:36.575912 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.575900    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c36c08b-8r5hc\" (UID: \"c422eb47-48de-4d6c-bc9d-812797f4cb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:36.576399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.576378    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c36c08b-8r5hc\" (UID: \"c422eb47-48de-4d6c-bc9d-812797f4cb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:36.578247 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.578228    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c36c08b-8r5hc\" (UID: \"c422eb47-48de-4d6c-bc9d-812797f4cb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:36.633363 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.633274    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2gn\" (UniqueName: \"kubernetes.io/projected/c422eb47-48de-4d6c-bc9d-812797f4cb26-kube-api-access-2h2gn\") pod \"maas-api-revoke-sub-1c36c08b-8r5hc\" (UID: \"c422eb47-48de-4d6c-bc9d-812797f4cb26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:36.691636 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:36.691606    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:37.073918 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:37.073872    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"]
Oct 31 18:40:37.197740 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:37.197657    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" event={"ID":"8fd5dbc4-d25a-4018-866e-2161b2378eaa","Type":"ContainerStarted","Data":"443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518"}
Oct 31 18:40:37.197740 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:37.197714    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" event={"ID":"8fd5dbc4-d25a-4018-866e-2161b2378eaa","Type":"ContainerStarted","Data":"895f6ff07e8e84390e4845c6cb045746869a8ed02430d7d97f886ae8ac91217a"}
Oct 31 18:40:37.199594 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:37.199564    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc" event={"ID":"c422eb47-48de-4d6c-bc9d-812797f4cb26","Type":"ContainerStarted","Data":"1bb8d6f7078dd9dfb9b3c5833f7dd13a2f108d8697b475463e65e495e442701d"}
Oct 31 18:40:37.318105 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:37.318057    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" podStartSLOduration=2.318043606 podStartE2EDuration="2.318043606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:37.315548019 +0000 UTC m=+1183.461997885" watchObservedRunningTime="2026-10-31 18:40:37.318043606 +0000 UTC m=+1183.464493464"
Oct 31 18:40:38.082805 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:38.082770    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:38.086627 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:38.086594    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 18:40:38.086769 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:38.086644    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" podUID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 18:40:38.207258 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:38.207228    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc" event={"ID":"c422eb47-48de-4d6c-bc9d-812797f4cb26","Type":"ContainerStarted","Data":"34c456b8de3e4c00faaa2160ee3792f6879e799e3a3be84b20d345ea7b768e19"}
Oct 31 18:40:38.348135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:38.348040    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc" podStartSLOduration=2.348026817 podStartE2EDuration="2.348026817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:40:38.344666765 +0000 UTC m=+1184.491116633" watchObservedRunningTime="2026-10-31 18:40:38.348026817 +0000 UTC m=+1184.494476680"
Oct 31 18:40:38.712048 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:38.711942    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:39.081910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:39.081872    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 18:40:39.082207 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:39.082178    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" podUID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 18:40:39.217015 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:39.216983    4484 generic.go:356] "Generic (PLEG): container finished" podID="c422eb47-48de-4d6c-bc9d-812797f4cb26" containerID="34c456b8de3e4c00faaa2160ee3792f6879e799e3a3be84b20d345ea7b768e19" exitCode=0
Oct 31 18:40:39.217410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:39.217046    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc" event={"ID":"c422eb47-48de-4d6c-bc9d-812797f4cb26","Type":"ContainerDied","Data":"34c456b8de3e4c00faaa2160ee3792f6879e799e3a3be84b20d345ea7b768e19"}
Oct 31 18:40:40.086062 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.085889    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 18:40:40.086062 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.085967    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" podUID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 18:40:40.417534 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.417507    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:40.430567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.430541    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c422eb47-48de-4d6c-bc9d-812797f4cb26-kube-api-access-2h2gn\" (UniqueName: \"kubernetes.io/projected/c422eb47-48de-4d6c-bc9d-812797f4cb26-kube-api-access-2h2gn\") pod \"c422eb47-48de-4d6c-bc9d-812797f4cb26\" (UID: \"c422eb47-48de-4d6c-bc9d-812797f4cb26\") "
Oct 31 18:40:40.430704 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.430676    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-openshift-ca\") pod \"c422eb47-48de-4d6c-bc9d-812797f4cb26\" (UID: \"c422eb47-48de-4d6c-bc9d-812797f4cb26\") "
Oct 31 18:40:40.430774 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.430715    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-xks-ca\") pod \"c422eb47-48de-4d6c-bc9d-812797f4cb26\" (UID: \"c422eb47-48de-4d6c-bc9d-812797f4cb26\") "
Oct 31 18:40:40.431070 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.431043    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-openshift-ca" pod "c422eb47-48de-4d6c-bc9d-812797f4cb26" (UID: "c422eb47-48de-4d6c-bc9d-812797f4cb26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:40:40.436814 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.436786    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-xks-ca" pod "c422eb47-48de-4d6c-bc9d-812797f4cb26" (UID: "c422eb47-48de-4d6c-bc9d-812797f4cb26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:40:40.436906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.436847    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c422eb47-48de-4d6c-bc9d-812797f4cb26-kube-api-access-2h2gn" pod "c422eb47-48de-4d6c-bc9d-812797f4cb26" (UID: "c422eb47-48de-4d6c-bc9d-812797f4cb26"). InnerVolumeSpecName "kube-api-access-2h2gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:40:40.505216 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.505182    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:40.532178 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.532125    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:40.532178 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.532160    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c422eb47-48de-4d6c-bc9d-812797f4cb26-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:40.532178 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:40.532176    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h2gn\" (UniqueName: \"kubernetes.io/projected/c422eb47-48de-4d6c-bc9d-812797f4cb26-kube-api-access-2h2gn\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:41.085956 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:41.085915    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 18:40:41.086138 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:41.085994    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" podUID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 18:40:41.230610 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:41.230581    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"
Oct 31 18:40:41.230752 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:41.230604    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc" event={"ID":"c422eb47-48de-4d6c-bc9d-812797f4cb26","Type":"ContainerDied","Data":"1bb8d6f7078dd9dfb9b3c5833f7dd13a2f108d8697b475463e65e495e442701d"}
Oct 31 18:40:41.230752 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:41.230653    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bb8d6f7078dd9dfb9b3c5833f7dd13a2f108d8697b475463e65e495e442701d"
Oct 31 18:40:42.084499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:42.080886    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 18:40:42.084499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:42.080957    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" podUID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 18:40:43.080676 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:43.080636    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 31 18:40:43.080881 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:43.080707    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" podUID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 31 18:40:43.381357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:43.381271    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:44.085110 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:44.085079    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:44.085450 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:44.085432    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:44.086125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:44.086106    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:40:46.776076 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:46.776029    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:48.078031 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:48.078000    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:51.370400 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:51.370370    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:40:51.385770 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:51.385736    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:52.306242 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.306210    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"]
Oct 31 18:40:52.307538 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.307504    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c422eb47-48de-4d6c-bc9d-812797f4cb26" containerName="revoke-keys"
Oct 31 18:40:52.307697 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.307680    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c422eb47-48de-4d6c-bc9d-812797f4cb26" containerName="revoke-keys"
Oct 31 18:40:52.307954 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.307938    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c422eb47-48de-4d6c-bc9d-812797f4cb26" containerName="revoke-keys"
Oct 31 18:40:52.312254 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.312236    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:52.362976 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.362939    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"]
Oct 31 18:40:52.380451 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.380411    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd85448-6da1-473e-857e-d02837627368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84ee3021-cqcnn\" (UID: \"ccd85448-6da1-473e-857e-d02837627368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:52.382053 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.382025    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd85448-6da1-473e-857e-d02837627368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84ee3021-cqcnn\" (UID: \"ccd85448-6da1-473e-857e-d02837627368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:52.382301 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.382282    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2d7m\" (UniqueName: \"kubernetes.io/projected/ccd85448-6da1-473e-857e-d02837627368-kube-api-access-q2d7m\") pod \"maas-api-revoke-sub-84ee3021-cqcnn\" (UID: \"ccd85448-6da1-473e-857e-d02837627368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:52.417658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.417633    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:40:52.419340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.417459    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:40:52.511500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.487385    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd85448-6da1-473e-857e-d02837627368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84ee3021-cqcnn\" (UID: \"ccd85448-6da1-473e-857e-d02837627368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:52.511500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.487539    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd85448-6da1-473e-857e-d02837627368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84ee3021-cqcnn\" (UID: \"ccd85448-6da1-473e-857e-d02837627368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:52.511500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.487588    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2d7m\" (UniqueName: \"kubernetes.io/projected/ccd85448-6da1-473e-857e-d02837627368-kube-api-access-q2d7m\") pod \"maas-api-revoke-sub-84ee3021-cqcnn\" (UID: \"ccd85448-6da1-473e-857e-d02837627368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:52.575551 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.568285    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd85448-6da1-473e-857e-d02837627368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84ee3021-cqcnn\" (UID: \"ccd85448-6da1-473e-857e-d02837627368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:52.592502 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.591936    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd85448-6da1-473e-857e-d02837627368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84ee3021-cqcnn\" (UID: \"ccd85448-6da1-473e-857e-d02837627368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:52.616119 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.615292    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2d7m\" (UniqueName: \"kubernetes.io/projected/ccd85448-6da1-473e-857e-d02837627368-kube-api-access-q2d7m\") pod \"maas-api-revoke-sub-84ee3021-cqcnn\" (UID: \"ccd85448-6da1-473e-857e-d02837627368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:52.633707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:52.633671    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:53.203188 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:53.202350    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:40:53.822818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:53.817699    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"]
Oct 31 18:40:53.908975 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:53.894771    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6" podUID="a91dc442-830a-4436-bab0-fedc94f75038" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:40:54.165637 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:54.165582    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr" podUID="020ad937-bef8-41ae-9394-eab2f30eefba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:40:54.227563 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:54.226565    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4" podUID="7fdc08a1-c720-4038-acca-b931fbec7633" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:40:54.358895 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:54.358868    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:40:54.391128 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:54.391103    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:40:54.392935 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:54.392904    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn" event={"ID":"ccd85448-6da1-473e-857e-d02837627368","Type":"ContainerStarted","Data":"4af06bb722226ed8d55a8d9c9ffc62e6ed3554030e25bbf073512e9de474d45f"}
Oct 31 18:40:55.400592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:55.400503    4484 generic.go:356] "Generic (PLEG): container finished" podID="ccd85448-6da1-473e-857e-d02837627368" containerID="b7f5bd7c70a24a496280bafd4bc8fcf979f1790c43c6a24fb6ed01b65929eb59" exitCode=0
Oct 31 18:40:55.401034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:55.400642    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn" event={"ID":"ccd85448-6da1-473e-857e-d02837627368","Type":"ContainerDied","Data":"b7f5bd7c70a24a496280bafd4bc8fcf979f1790c43c6a24fb6ed01b65929eb59"}
Oct 31 18:40:55.469007 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:40:55.468973    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:40:56.552509 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:56.552457    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:56.670172 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:56.670091    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd85448-6da1-473e-857e-d02837627368-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd85448-6da1-473e-857e-d02837627368-maas-api-xks-ca\") pod \"ccd85448-6da1-473e-857e-d02837627368\" (UID: \"ccd85448-6da1-473e-857e-d02837627368\") "
Oct 31 18:40:56.670325 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:56.670249    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd85448-6da1-473e-857e-d02837627368-kube-api-access-q2d7m\" (UniqueName: \"kubernetes.io/projected/ccd85448-6da1-473e-857e-d02837627368-kube-api-access-q2d7m\") pod \"ccd85448-6da1-473e-857e-d02837627368\" (UID: \"ccd85448-6da1-473e-857e-d02837627368\") "
Oct 31 18:40:56.670325 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:56.670298    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd85448-6da1-473e-857e-d02837627368-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd85448-6da1-473e-857e-d02837627368-maas-api-openshift-ca\") pod \"ccd85448-6da1-473e-857e-d02837627368\" (UID: \"ccd85448-6da1-473e-857e-d02837627368\") "
Oct 31 18:40:56.670864 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:56.670834    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd85448-6da1-473e-857e-d02837627368-maas-api-openshift-ca" pod "ccd85448-6da1-473e-857e-d02837627368" (UID: "ccd85448-6da1-473e-857e-d02837627368"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:40:56.672907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:56.672882    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd85448-6da1-473e-857e-d02837627368-kube-api-access-q2d7m" pod "ccd85448-6da1-473e-857e-d02837627368" (UID: "ccd85448-6da1-473e-857e-d02837627368"). InnerVolumeSpecName "kube-api-access-q2d7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:40:56.672907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:56.672886    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd85448-6da1-473e-857e-d02837627368-maas-api-xks-ca" pod "ccd85448-6da1-473e-857e-d02837627368" (UID: "ccd85448-6da1-473e-857e-d02837627368"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:40:56.772122 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:56.772096    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd85448-6da1-473e-857e-d02837627368-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:56.772122 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:56.772118    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2d7m\" (UniqueName: \"kubernetes.io/projected/ccd85448-6da1-473e-857e-d02837627368-kube-api-access-q2d7m\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:56.772301 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:56.772130    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd85448-6da1-473e-857e-d02837627368-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:40:57.413143 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:57.413113    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"
Oct 31 18:40:57.413322 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:57.413157    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn" event={"ID":"ccd85448-6da1-473e-857e-d02837627368","Type":"ContainerDied","Data":"4af06bb722226ed8d55a8d9c9ffc62e6ed3554030e25bbf073512e9de474d45f"}
Oct 31 18:40:57.413322 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:57.413200    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4af06bb722226ed8d55a8d9c9ffc62e6ed3554030e25bbf073512e9de474d45f"
Oct 31 18:40:58.044370 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:40:58.044327    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:02.662347 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:02.662286    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:03.098399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.098362    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"]
Oct 31 18:41:03.098933 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.098921    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ccd85448-6da1-473e-857e-d02837627368" containerName="revoke-keys"
Oct 31 18:41:03.098986 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.098935    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd85448-6da1-473e-857e-d02837627368" containerName="revoke-keys"
Oct 31 18:41:03.099034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.099026    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd85448-6da1-473e-857e-d02837627368" containerName="revoke-keys"
Oct 31 18:41:03.102650 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.102629    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:03.159221 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.159190    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"]
Oct 31 18:41:03.205536 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.205502    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:41:03.205725 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.205591    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:41:03.243378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.243348    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmm5\" (UniqueName: \"kubernetes.io/projected/40380b55-67f1-4f80-9152-f449137dce80-kube-api-access-rtmm5\") pod \"maas-api-revoke-sub-c9a2fda1-gwk7j\" (UID: \"40380b55-67f1-4f80-9152-f449137dce80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:03.243567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.243396    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40380b55-67f1-4f80-9152-f449137dce80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a2fda1-gwk7j\" (UID: \"40380b55-67f1-4f80-9152-f449137dce80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:03.243617 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.243577    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40380b55-67f1-4f80-9152-f449137dce80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a2fda1-gwk7j\" (UID: \"40380b55-67f1-4f80-9152-f449137dce80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:03.270152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.270127    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"]
Oct 31 18:41:03.274070 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.274047    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.327457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.327425    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-da6ea7-openshift-default-dockercfg-z7k66\""
Oct 31 18:41:03.331128 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.331103    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"]
Oct 31 18:41:03.345171 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.345143    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlx22\" (UniqueName: \"kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-kube-api-access-mlx22\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.345336 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.345207    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40380b55-67f1-4f80-9152-f449137dce80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a2fda1-gwk7j\" (UID: \"40380b55-67f1-4f80-9152-f449137dce80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:03.345336 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.345283    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.345336 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.345319    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.345457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.345398    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.345457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.345450    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmm5\" (UniqueName: \"kubernetes.io/projected/40380b55-67f1-4f80-9152-f449137dce80-kube-api-access-rtmm5\") pod \"maas-api-revoke-sub-c9a2fda1-gwk7j\" (UID: \"40380b55-67f1-4f80-9152-f449137dce80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:03.345592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.345499    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.345592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.345532    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40380b55-67f1-4f80-9152-f449137dce80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a2fda1-gwk7j\" (UID: \"40380b55-67f1-4f80-9152-f449137dce80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:03.345592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.345549    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.345748 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.345687    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.345748 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.345739    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.345846 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.345776    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.346090 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.346065    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40380b55-67f1-4f80-9152-f449137dce80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a2fda1-gwk7j\" (UID: \"40380b55-67f1-4f80-9152-f449137dce80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:03.348138 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.348118    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40380b55-67f1-4f80-9152-f449137dce80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a2fda1-gwk7j\" (UID: \"40380b55-67f1-4f80-9152-f449137dce80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:03.403183 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.403118    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmm5\" (UniqueName: \"kubernetes.io/projected/40380b55-67f1-4f80-9152-f449137dce80-kube-api-access-rtmm5\") pod \"maas-api-revoke-sub-c9a2fda1-gwk7j\" (UID: \"40380b55-67f1-4f80-9152-f449137dce80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:03.415748 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.415722    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:03.447076 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.447040    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.447236 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.447101    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.447236 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.447134    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.447236 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.447190    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlx22\" (UniqueName: \"kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-kube-api-access-mlx22\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.447674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.447433    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.447674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.447465    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.447674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.447509    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.447674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.447592    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.447674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.447651    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.447977 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.447692    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.448072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.448055    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.448132 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.448066    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.448359 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.448327    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.448615 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.448360    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.454398 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.454366    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.456014 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.455989    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.507019 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.506983    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.507978 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.507945    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlx22\" (UniqueName: \"kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-kube-api-access-mlx22\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.587093 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.587054    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:03.794304 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.794277    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"]
Oct 31 18:41:03.794679 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:41:03.794501    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40380b55_67f1_4f80_9152_f449137dce80.slice/crio-398c37c3c86ecf945dc76db3edf7e89da65a873d4fdf78f7cded5dfb8d8f9d61 WatchSource:0}: Error finding container 398c37c3c86ecf945dc76db3edf7e89da65a873d4fdf78f7cded5dfb8d8f9d61: Status 404 returned error can't find the container with id 398c37c3c86ecf945dc76db3edf7e89da65a873d4fdf78f7cded5dfb8d8f9d61
Oct 31 18:41:03.960453 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.960424    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"]
Oct 31 18:41:03.960825 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:41:03.960799    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc48784_e2e5_4d2f_90f0_9edbe9c9644f.slice/crio-4df1b658c8ab12e37a7237660f263685e22dfd8441366dea0b756978b49e9e1f WatchSource:0}: Error finding container 4df1b658c8ab12e37a7237660f263685e22dfd8441366dea0b756978b49e9e1f: Status 404 returned error can't find the container with id 4df1b658c8ab12e37a7237660f263685e22dfd8441366dea0b756978b49e9e1f
Oct 31 18:41:03.963666 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.963634    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:41:03.963742 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.963697    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:41:03.963742 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:03.963725    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:41:04.461563 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:04.461111    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" event={"ID":"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f","Type":"ContainerStarted","Data":"0a06e241445c11c5c98dd80787ae60f68668d565e813a1db0cb78e1b75e63394"}
Oct 31 18:41:04.461563 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:04.461248    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" event={"ID":"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f","Type":"ContainerStarted","Data":"4df1b658c8ab12e37a7237660f263685e22dfd8441366dea0b756978b49e9e1f"}
Oct 31 18:41:04.471635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:04.471592    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j" event={"ID":"40380b55-67f1-4f80-9152-f449137dce80","Type":"ContainerStarted","Data":"167753de7fc809b5e46612c7e051a72bee8febac30fabddd620d61a9f8a80d76"}
Oct 31 18:41:04.472647 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:04.472623    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j" event={"ID":"40380b55-67f1-4f80-9152-f449137dce80","Type":"ContainerStarted","Data":"398c37c3c86ecf945dc76db3edf7e89da65a873d4fdf78f7cded5dfb8d8f9d61"}
Oct 31 18:41:04.588224 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:04.587414    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" podStartSLOduration=1.5873939400000001 podStartE2EDuration="1.58739394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:04.584295346 +0000 UTC m=+1210.730745210" watchObservedRunningTime="2026-10-31 18:41:04.58739394 +0000 UTC m=+1210.733843804"
Oct 31 18:41:04.588224 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:04.587785    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:04.700332 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:04.700099    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j" podStartSLOduration=1.700081828 podStartE2EDuration="1.700081828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:04.697729409 +0000 UTC m=+1210.844179285" watchObservedRunningTime="2026-10-31 18:41:04.700081828 +0000 UTC m=+1210.846531691"
Oct 31 18:41:05.490419 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:05.490382    4484 generic.go:356] "Generic (PLEG): container finished" podID="40380b55-67f1-4f80-9152-f449137dce80" containerID="167753de7fc809b5e46612c7e051a72bee8febac30fabddd620d61a9f8a80d76" exitCode=0
Oct 31 18:41:05.490854 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:05.490485    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j" event={"ID":"40380b55-67f1-4f80-9152-f449137dce80","Type":"ContainerDied","Data":"167753de7fc809b5e46612c7e051a72bee8febac30fabddd620d61a9f8a80d76"}
Oct 31 18:41:05.592833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:05.592795    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 18:41:05.593008 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:05.592854    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" podUID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 18:41:06.591116 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:06.591083    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 18:41:06.592116 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:06.591715    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" podUID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 18:41:06.659541 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:06.659518    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:06.809759 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:06.809668    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40380b55-67f1-4f80-9152-f449137dce80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40380b55-67f1-4f80-9152-f449137dce80-maas-api-openshift-ca\") pod \"40380b55-67f1-4f80-9152-f449137dce80\" (UID: \"40380b55-67f1-4f80-9152-f449137dce80\") "
Oct 31 18:41:06.809759 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:06.809717    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40380b55-67f1-4f80-9152-f449137dce80-kube-api-access-rtmm5\" (UniqueName: \"kubernetes.io/projected/40380b55-67f1-4f80-9152-f449137dce80-kube-api-access-rtmm5\") pod \"40380b55-67f1-4f80-9152-f449137dce80\" (UID: \"40380b55-67f1-4f80-9152-f449137dce80\") "
Oct 31 18:41:06.809759 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:06.809752    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40380b55-67f1-4f80-9152-f449137dce80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40380b55-67f1-4f80-9152-f449137dce80-maas-api-xks-ca\") pod \"40380b55-67f1-4f80-9152-f449137dce80\" (UID: \"40380b55-67f1-4f80-9152-f449137dce80\") "
Oct 31 18:41:06.810095 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:06.810066    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40380b55-67f1-4f80-9152-f449137dce80-maas-api-openshift-ca" pod "40380b55-67f1-4f80-9152-f449137dce80" (UID: "40380b55-67f1-4f80-9152-f449137dce80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:41:06.810248 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:06.810231    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40380b55-67f1-4f80-9152-f449137dce80-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:41:06.812080 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:06.812043    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40380b55-67f1-4f80-9152-f449137dce80-maas-api-xks-ca" pod "40380b55-67f1-4f80-9152-f449137dce80" (UID: "40380b55-67f1-4f80-9152-f449137dce80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:41:06.812178 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:06.812131    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40380b55-67f1-4f80-9152-f449137dce80-kube-api-access-rtmm5" pod "40380b55-67f1-4f80-9152-f449137dce80" (UID: "40380b55-67f1-4f80-9152-f449137dce80"). InnerVolumeSpecName "kube-api-access-rtmm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:41:06.911289 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:06.911261    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmm5\" (UniqueName: \"kubernetes.io/projected/40380b55-67f1-4f80-9152-f449137dce80-kube-api-access-rtmm5\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:41:06.911289 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:06.911286    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40380b55-67f1-4f80-9152-f449137dce80-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:41:07.407254 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:07.407213    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:07.505457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:07.505426    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"
Oct 31 18:41:07.505457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:07.505442    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j" event={"ID":"40380b55-67f1-4f80-9152-f449137dce80","Type":"ContainerDied","Data":"398c37c3c86ecf945dc76db3edf7e89da65a873d4fdf78f7cded5dfb8d8f9d61"}
Oct 31 18:41:07.505717 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:07.505508    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="398c37c3c86ecf945dc76db3edf7e89da65a873d4fdf78f7cded5dfb8d8f9d61"
Oct 31 18:41:07.587933 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:07.587904    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 18:41:07.588104 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:07.587953    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" podUID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 18:41:08.588501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:08.588362    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 18:41:08.588501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:08.588431    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" podUID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 18:41:09.048280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:09.048246    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"]
Oct 31 18:41:09.098572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:09.098539    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b82135e3-cqq8k"]
Oct 31 18:41:09.587877 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:09.587846    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 31 18:41:09.588100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:09.587906    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" podUID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 31 18:41:09.996250 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:09.996209    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ae0eac-eaeb-4a7e-92fe-0606deee7ff2" path="/var/lib/kubelet/pods/30ae0eac-eaeb-4a7e-92fe-0606deee7ff2/volumes"
Oct 31 18:41:11.588786 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:11.588751    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:41:11.589464 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:11.589432    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" podUID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:12.353500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:12.343462    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:12.634554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:12.633584    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:41:12.634554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:12.633670    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" podUID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:12.969929 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:12.969839    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:12.972887 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:12.972865    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:13.009780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:13.009754    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:41:13.081499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:13.078560    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4" podUID="7fdc08a1-c720-4038-acca-b931fbec7633" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:15.811429 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:15.811391    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:16.479438 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:16.479397    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:16.678899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:16.678863    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:17.268398 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:17.268363    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:17.738443 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:17.738406    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:17.917979 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:17.917942    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:21.125908 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.125878    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"]
Oct 31 18:41:21.126419 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.126407    4484 cpu_manager.go:414] "containerMap: removing container" podUID="40380b55-67f1-4f80-9152-f449137dce80" containerName="revoke-keys"
Oct 31 18:41:21.126465 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.126420    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40380b55-67f1-4f80-9152-f449137dce80" containerName="revoke-keys"
Oct 31 18:41:21.126555 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.126545    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="40380b55-67f1-4f80-9152-f449137dce80" containerName="revoke-keys"
Oct 31 18:41:21.129973 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.129956    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.162626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.162604    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.162748 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.162657    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.162748 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.162715    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.162835 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.162765    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw9x4\" (UniqueName: \"kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-kube-api-access-sw9x4\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.162876 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.162848    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.162910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.162884    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4415ab7-0e82-435f-b701-544af2b06359-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.162972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.162954    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.163010 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.162992    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4415ab7-0e82-435f-b701-544af2b06359-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.163051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.163037    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.236422 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.236394    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"]
Oct 31 18:41:21.275781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.275742    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.275962 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.275820    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.275962 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.275848    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.275962 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.275893    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw9x4\" (UniqueName: \"kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-kube-api-access-sw9x4\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.275962 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.275924    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.276192 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.275974    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4415ab7-0e82-435f-b701-544af2b06359-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.276192 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.276054    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.276192 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.276110    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4415ab7-0e82-435f-b701-544af2b06359-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.276192 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.276183    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.276609 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.276584    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.276823 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.276697    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.277571 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.277504    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.277571 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.277542    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4415ab7-0e82-435f-b701-544af2b06359-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.278081 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.278062    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.279233 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.279207    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.280182 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.280154    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4415ab7-0e82-435f-b701-544af2b06359-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.333482 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.333450    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.334380 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.334359    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw9x4\" (UniqueName: \"kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-kube-api-access-sw9x4\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.444396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.444327    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:21.926952 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.926926    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"]
Oct 31 18:41:21.927978 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:41:21.927938    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4415ab7_0e82_435f_b701_544af2b06359.slice/crio-9b0673d28ac4709199d6903e354a44c4c98c94c72acb231ec84f306eb7ce9080 WatchSource:0}: Error finding container 9b0673d28ac4709199d6903e354a44c4c98c94c72acb231ec84f306eb7ce9080: Status 404 returned error can't find the container with id 9b0673d28ac4709199d6903e354a44c4c98c94c72acb231ec84f306eb7ce9080
Oct 31 18:41:21.930323 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.930291    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:41:21.930409 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.930356    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:41:21.930409 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:21.930386    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:41:22.115997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:22.115957    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:22.657745 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:22.657712    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" event={"ID":"a4415ab7-0e82-435f-b701-544af2b06359","Type":"ContainerStarted","Data":"c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5"}
Oct 31 18:41:22.658136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:22.657769    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" event={"ID":"a4415ab7-0e82-435f-b701-544af2b06359","Type":"ContainerStarted","Data":"9b0673d28ac4709199d6903e354a44c4c98c94c72acb231ec84f306eb7ce9080"}
Oct 31 18:41:22.830269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:22.830216    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" podStartSLOduration=1.830202358 podStartE2EDuration="1.830202358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:22.828095204 +0000 UTC m=+1228.974545068" watchObservedRunningTime="2026-10-31 18:41:22.830202358 +0000 UTC m=+1228.976652292"
Oct 31 18:41:23.444870 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:23.444839    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:23.446020 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:23.445998    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 18:41:23.446153 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:23.446040    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" podUID="a4415ab7-0e82-435f-b701-544af2b06359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 18:41:24.445670 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:24.445636    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 18:41:24.446041 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:24.445694    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" podUID="a4415ab7-0e82-435f-b701-544af2b06359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 18:41:25.444787 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:25.444753    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 18:41:25.444970 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:25.444811    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" podUID="a4415ab7-0e82-435f-b701-544af2b06359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 18:41:25.494843 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:41:25.494806    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:41:26.445214 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:26.445181    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 18:41:26.445407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:26.445242    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" podUID="a4415ab7-0e82-435f-b701-544af2b06359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 18:41:26.986873 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:26.986836    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:27.444778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:27.444691    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 18:41:27.444778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:27.444743    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" podUID="a4415ab7-0e82-435f-b701-544af2b06359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 18:41:28.444925 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:28.444891    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 18:41:28.445307 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:28.444955    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" podUID="a4415ab7-0e82-435f-b701-544af2b06359" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 18:41:29.449572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:29.449544    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:29.450025 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:29.450012    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:29.450690 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:29.450669    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:41:31.691646 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:31.691615    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:32.377210 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:32.377181    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:41:34.133498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:34.123718    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr" podUID="020ad937-bef8-41ae-9394-eab2f30eefba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:34.196500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:34.195648    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84" podUID="a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:34.271951 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:34.271613    4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6" podUID="a91dc442-830a-4436-bab0-fedc94f75038" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:34.353500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:34.330128    4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr" podUID="020ad937-bef8-41ae-9394-eab2f30eefba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:34.497609 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:34.497563    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c" podUID="307a9ca2-cb52-4131-b750-2fb5d89e8235" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:34.711669 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:34.711614    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv" podUID="3b6b07cc-6154-4582-8e24-fc085949657a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:34.861681 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:34.861572    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6" podUID="a91dc442-830a-4436-bab0-fedc94f75038" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:35.027617 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:35.027567    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4" podUID="7fdc08a1-c720-4038-acca-b931fbec7633" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:35.127570 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:35.126562    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr" podUID="020ad937-bef8-41ae-9394-eab2f30eefba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:35.127570 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:35.127318    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:41:35.613546 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:35.595564    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c" podUID="307a9ca2-cb52-4131-b750-2fb5d89e8235" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:35.613546 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:35.595984    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:41:35.613546 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:35.604085    4484 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84" podUID="a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded"
Oct 31 18:41:35.714882 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:35.713589    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv" podUID="3b6b07cc-6154-4582-8e24-fc085949657a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:35.714882 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:35.714055    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:41:35.800492 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:35.800423    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:35.871565 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:35.869857    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6" podUID="a91dc442-830a-4436-bab0-fedc94f75038" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:35.871565 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:35.871041    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:41:36.077496 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:36.067010    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4" podUID="7fdc08a1-c720-4038-acca-b931fbec7633" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 31 18:41:36.130750 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:36.130631    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr" podUID="020ad937-bef8-41ae-9394-eab2f30eefba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 31 18:41:36.285526 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:36.285490    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:41:36.285785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:36.285754    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:36.367612 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:36.367575    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 18:41:36.367804 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:36.367673    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:41:36.367804 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:36.367732    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 18:41:36.367804 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:36.367784    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 18:41:36.596318 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:36.596292    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 18:41:36.714313 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:36.714277    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv" podUID="3b6b07cc-6154-4582-8e24-fc085949657a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:41:37.207998 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:37.205277    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 18:41:38.288501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:38.288425    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:41.486399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:41.486364    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:43.064015 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:43.063985    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:43.993757 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:43.993719    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:44.673256 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:44.673221    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:44.948547 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:44.948432    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:47.117749 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:47.117715    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:47.865175 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:47.865144    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:48.864210 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:48.864168    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:49.374644 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:49.374617    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:49.551357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:49.551322    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:49.715671 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:49.715575    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:51.753746 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:51.753705    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:52.547094 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:52.547063    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:52.729141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:52.729094    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:52.936639 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:52.936554    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"]
Oct 31 18:41:52.941734 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:52.941712    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"
Oct 31 18:41:52.983186 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:52.983157    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:52.990555 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:52.990524    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2gr\" (UniqueName: \"kubernetes.io/projected/97aced5d-a853-4858-bbe7-fbe1088dd7fe-kube-api-access-8r2gr\") pod \"payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l\" (UID: \"97aced5d-a853-4858-bbe7-fbe1088dd7fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"
Oct 31 18:41:52.990698 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:52.990623    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97aced5d-a853-4858-bbe7-fbe1088dd7fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l\" (UID: \"97aced5d-a853-4858-bbe7-fbe1088dd7fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"
Oct 31 18:41:52.994352 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:52.994332    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3417b822\""
Oct 31 18:41:53.046356 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:53.046313    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3417b822-dockercfg-dgl9j\""
Oct 31 18:41:53.050259 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:53.050223    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"]
Oct 31 18:41:53.105054 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:53.105013    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97aced5d-a853-4858-bbe7-fbe1088dd7fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l\" (UID: \"97aced5d-a853-4858-bbe7-fbe1088dd7fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"
Oct 31 18:41:53.105337 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:53.105314    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2gr\" (UniqueName: \"kubernetes.io/projected/97aced5d-a853-4858-bbe7-fbe1088dd7fe-kube-api-access-8r2gr\") pod \"payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l\" (UID: \"97aced5d-a853-4858-bbe7-fbe1088dd7fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"
Oct 31 18:41:53.106772 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:53.106746    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97aced5d-a853-4858-bbe7-fbe1088dd7fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l\" (UID: \"97aced5d-a853-4858-bbe7-fbe1088dd7fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"
Oct 31 18:41:53.162948 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:53.162919    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2gr\" (UniqueName: \"kubernetes.io/projected/97aced5d-a853-4858-bbe7-fbe1088dd7fe-kube-api-access-8r2gr\") pod \"payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l\" (UID: \"97aced5d-a853-4858-bbe7-fbe1088dd7fe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"
Oct 31 18:41:53.257940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:53.257906    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"
Oct 31 18:41:53.632687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:53.632662    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"]
Oct 31 18:41:53.633004 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:41:53.632982    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97aced5d_a853_4858_bbe7_fbe1088dd7fe.slice/crio-be44a99fb4d665a338af77e9d6773492586abd6b62ade16774636d0f14dbfdd2 WatchSource:0}: Error finding container be44a99fb4d665a338af77e9d6773492586abd6b62ade16774636d0f14dbfdd2: Status 404 returned error can't find the container with id be44a99fb4d665a338af77e9d6773492586abd6b62ade16774636d0f14dbfdd2
Oct 31 18:41:54.159029 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:54.159000    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l" event={"ID":"97aced5d-a853-4858-bbe7-fbe1088dd7fe","Type":"ContainerStarted","Data":"8c9f188a878e878dafc883dced60d8fd197a7800bd96bb81edd127e3fbb0e42d"}
Oct 31 18:41:54.159417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:54.159056    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l" event={"ID":"97aced5d-a853-4858-bbe7-fbe1088dd7fe","Type":"ContainerStarted","Data":"be44a99fb4d665a338af77e9d6773492586abd6b62ade16774636d0f14dbfdd2"}
Oct 31 18:41:54.159417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:54.159195    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"
Oct 31 18:41:54.274372 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:54.274323    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l" podStartSLOduration=2.274309243 podStartE2EDuration="2.274309243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:41:54.272549196 +0000 UTC m=+1260.418999072" watchObservedRunningTime="2026-10-31 18:41:54.274309243 +0000 UTC m=+1260.420759107"
Oct 31 18:41:55.468382 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:41:55.468346    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:41:57.655352 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:57.655320    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:41:58.399514 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.399452    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"]
Oct 31 18:41:58.431321 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.431292    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.480233 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.480173    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.480586 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.480560    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.480739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.480720    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.480907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.480889    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.481031 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.481012    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bqwz\" (UniqueName: \"kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-kube-api-access-8bqwz\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.481247 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.481229    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.481383 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.481366    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.481552 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.481534    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.481729 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.481710    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.535938 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.535903    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"]
Oct 31 18:41:58.626501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.607008    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.627779 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.627753    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.628039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.628018    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bqwz\" (UniqueName: \"kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-kube-api-access-8bqwz\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.629555 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.629513    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.630220 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.630202    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.641537 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.630127    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.641967 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.630550    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.642359 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.642339    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.642710 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.642597    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.680261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.642638    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.680261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.678693    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.680261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.678858    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.680261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.679279    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.680261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.679530    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.828859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.822320    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.856520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.829912    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bqwz\" (UniqueName: \"kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-kube-api-access-8bqwz\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:58.856520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:58.830071    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:59.023419 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:59.023354    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:59.116500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:59.112848    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:41:59.932063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:59.932033    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"]
Oct 31 18:41:59.951758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:59.950747    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:41:59.951758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:59.950835    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:41:59.951758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:41:59.950904    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:42:00.260063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:00.258820    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" event={"ID":"3e4b3792-3dcd-4def-972e-a2876ecc56cd","Type":"ContainerStarted","Data":"9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2"}
Oct 31 18:42:00.260063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:00.258888    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" event={"ID":"3e4b3792-3dcd-4def-972e-a2876ecc56cd","Type":"ContainerStarted","Data":"719afe8c7f60081e7691b7067118ff366f0f616d6e6fe515a35b3877321cd3e4"}
Oct 31 18:42:00.394500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:00.388521    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" podStartSLOduration=2.38850201 podStartE2EDuration="2.38850201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:00.38793655 +0000 UTC m=+1266.534386415" watchObservedRunningTime="2026-10-31 18:42:00.38850201 +0000 UTC m=+1266.534951875"
Oct 31 18:42:00.810498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:00.809666    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz57f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 18:42:00.810498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:00.809758    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" podUID="76199531-8025-41d1-813a-cc1a78125e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:01.122202 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:01.122095    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:42:01.122890 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:01.122866    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 18:42:01.123061 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:01.123036    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" podUID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 18:42:02.117925 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:02.117890    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 18:42:02.118107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:02.117956    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" podUID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 18:42:03.086520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.086453    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:03.119291 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.118875    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 18:42:03.119291 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.118947    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" podUID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 18:42:03.719707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.719672    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"]
Oct 31 18:42:03.723851 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.723834    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.814658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.814627    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.814658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.814663    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.814837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.814686    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.814837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.814804    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.814837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.814832    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpsdw\" (UniqueName: \"kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-kube-api-access-wpsdw\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.814959 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.814861    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.815011 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.814992    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.815051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.815039    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.815120 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.815104    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.825548 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.825522    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"]
Oct 31 18:42:03.916274 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.916237    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.916488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.916293    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.916488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.916316    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpsdw\" (UniqueName: \"kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-kube-api-access-wpsdw\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.916488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.916350    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.916488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.916435    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.916488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.916486    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.916743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.916537    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.916743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.916711    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.916854 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.916741    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.916854 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.916777    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.916957 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.916873    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.916957 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.916931    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.917125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.917105    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.917190 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.917112    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.918943 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.918925    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.919166 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.919148    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.988804 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.988751    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpsdw\" (UniqueName: \"kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-kube-api-access-wpsdw\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:03.989357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:03.989334    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:04.036518 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:04.036486    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:04.121895 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:04.121864    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 18:42:04.122324 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:04.121911    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" podUID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 18:42:04.493947 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:42:04.493919    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7cf4ebc_cd36_4baa_83df_2b2a9e485b9f.slice/crio-125eedbffcc16449951b9ca325f160ad78b0db29770b062ef1f64bdc91df74ab WatchSource:0}: Error finding container 125eedbffcc16449951b9ca325f160ad78b0db29770b062ef1f64bdc91df74ab: Status 404 returned error can't find the container with id 125eedbffcc16449951b9ca325f160ad78b0db29770b062ef1f64bdc91df74ab
Oct 31 18:42:04.494445 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:04.494417    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"]
Oct 31 18:42:04.496722 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:04.496693    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:42:04.496805 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:04.496756    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:42:04.496843 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:04.496786    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:42:05.117924 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:05.117886    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 18:42:05.118116 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:05.117955    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" podUID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 18:42:05.182674 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:05.182645    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"
Oct 31 18:42:05.520169 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:05.520140    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp" event={"ID":"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f","Type":"ContainerStarted","Data":"231b15f359fd930d52831d606f71f575e31459c1619462414fad41aca854b729"}
Oct 31 18:42:05.520392 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:05.520379    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp" event={"ID":"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f","Type":"ContainerStarted","Data":"125eedbffcc16449951b9ca325f160ad78b0db29770b062ef1f64bdc91df74ab"}
Oct 31 18:42:05.641165 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:05.641107    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp" podStartSLOduration=2.641093038 podStartE2EDuration="2.641093038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:05.639685189 +0000 UTC m=+1271.786135053" watchObservedRunningTime="2026-10-31 18:42:05.641093038 +0000 UTC m=+1271.787542946"
Oct 31 18:42:06.036765 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:06.036731    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:06.038340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:06.038314    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 18:42:06.038500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:06.038360    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp" podUID="d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 18:42:06.117680 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:06.117644    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 18:42:06.117884 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:06.117692    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" podUID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 18:42:07.037010 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:07.036980    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 18:42:07.037404 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:07.037033    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp" podUID="d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 18:42:07.117819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:07.117783    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 31 18:42:07.118003 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:07.117847    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" podUID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 31 18:42:07.855175 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:07.855143    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:08.037353 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:08.037325    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 18:42:08.037742 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:08.037373    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp" podUID="d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 18:42:08.121676 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:08.121588    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:42:08.121983 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:08.121964    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:42:08.122586 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:08.122568    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:42:09.036772 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:09.036739    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 18:42:09.036970 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:09.036795    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp" podUID="d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 18:42:10.037657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:10.037623    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 18:42:10.038048 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:10.037671    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp" podUID="d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 18:42:11.037829 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:11.037792    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 31 18:42:11.038271 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:11.037870    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp" podUID="d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 31 18:42:12.041436 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:12.041412    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:12.041859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:12.041823    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:12.042507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:12.042466    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:42:12.586781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:12.586749    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:17.119906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:17.119876    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:17.334463 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:17.334430    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:19.049063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:19.049030    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:42:21.221932 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:21.221898    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:21.376614 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:21.376580    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:22.230510 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.230457    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:22.231909 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.231876    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"]
Oct 31 18:42:22.236013 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.235989    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.292303 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.292251    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"]
Oct 31 18:42:22.337823 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.337761    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4ptj5\""
Oct 31 18:42:22.337823 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.337779    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 18:42:22.338099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.337832    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 18:42:22.338099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.337890    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ac4f996e253c\""
Oct 31 18:42:22.342886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.342798    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 18:42:22.342886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.342864    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 18:42:22.378432 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.378397    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-plugins-config-volume\") pod \"payload-processing-external-model-f55c59c9-p6k8m\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.378612 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.378463    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-provider-credentials\") pod \"payload-processing-external-model-f55c59c9-p6k8m\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.378664 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.378629    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-routing-overlay-volume\") pod \"payload-processing-external-model-f55c59c9-p6k8m\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.378701 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.378685    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm4ps\" (UniqueName: \"kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-kube-api-access-zm4ps\") pod \"payload-processing-external-model-f55c59c9-p6k8m\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.480207 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.480175    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-routing-overlay-volume\") pod \"payload-processing-external-model-f55c59c9-p6k8m\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.480390 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.480228    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm4ps\" (UniqueName: \"kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-kube-api-access-zm4ps\") pod \"payload-processing-external-model-f55c59c9-p6k8m\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.480390 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.480319    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-plugins-config-volume\") pod \"payload-processing-external-model-f55c59c9-p6k8m\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.480390 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.480352    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-provider-credentials\") pod \"payload-processing-external-model-f55c59c9-p6k8m\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.480972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.480916    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-routing-overlay-volume\") pod \"payload-processing-external-model-f55c59c9-p6k8m\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.481057 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.481009    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-plugins-config-volume\") pod \"payload-processing-external-model-f55c59c9-p6k8m\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.482804 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.482786    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-provider-credentials\") pod \"payload-processing-external-model-f55c59c9-p6k8m\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.538552 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.538530    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm4ps\" (UniqueName: \"kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-kube-api-access-zm4ps\") pod \"payload-processing-external-model-f55c59c9-p6k8m\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") " pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.550436 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.550412    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:22.933372 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:42:22.933327    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022ec6a8_e173_4976_b944_bcaef951597a.slice/crio-4270578dcadb1c9c830788e2f08f32a8ed2e57ae71dac84bee194e98501c60d6 WatchSource:0}: Error finding container 4270578dcadb1c9c830788e2f08f32a8ed2e57ae71dac84bee194e98501c60d6: Status 404 returned error can't find the container with id 4270578dcadb1c9c830788e2f08f32a8ed2e57ae71dac84bee194e98501c60d6
Oct 31 18:42:22.933549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:22.933530    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"]
Oct 31 18:42:23.665282 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:23.665246    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m" event={"ID":"022ec6a8-e173-4976-b944-bcaef951597a","Type":"ContainerStarted","Data":"61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb"}
Oct 31 18:42:23.665702 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:23.665302    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m" event={"ID":"022ec6a8-e173-4976-b944-bcaef951597a","Type":"ContainerStarted","Data":"4270578dcadb1c9c830788e2f08f32a8ed2e57ae71dac84bee194e98501c60d6"}
Oct 31 18:42:23.665702 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:23.665452    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:23.778997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:23.778952    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m" podStartSLOduration=1.778939615 podStartE2EDuration="1.778939615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:23.7786343 +0000 UTC m=+1289.925084163" watchObservedRunningTime="2026-10-31 18:42:23.778939615 +0000 UTC m=+1289.925389478"
Oct 31 18:42:25.468529 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:42:25.468466    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:42:26.950458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:26.950424    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:30.082716 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.082681    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:30.582141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.582105    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"]
Oct 31 18:42:30.587389 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.587374    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.654333 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.654300    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bd6c0a-openshift-default-dockercfg-zrhp5\""
Oct 31 18:42:30.680575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.680543    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-certs\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.680723 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.680591    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mkhn\" (UniqueName: \"kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-kube-api-access-2mkhn\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.680723 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.680690    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-envoy\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.680796 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.680756    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istiod-ca-cert\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.680796 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.680783    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-credential-socket\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.680870 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.680850    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-data\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.681001 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.680984    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-podinfo\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.681044 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.681014    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-token\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.681197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.681179    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-socket\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.686780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.686739    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:30.696363 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.696332    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"]
Oct 31 18:42:30.782606 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.782578    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-socket\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.782771 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.782628    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-certs\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.782771 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.782661    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mkhn\" (UniqueName: \"kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-kube-api-access-2mkhn\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.782771 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.782693    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-envoy\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.782771 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.782738    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istiod-ca-cert\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.782771 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.782765    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-credential-socket\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.783044 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.782785    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-data\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.783044 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.782965    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-podinfo\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.783044 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.783008    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-token\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.783199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.783069    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-socket\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.783199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.783139    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-certs\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.783307 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.783240    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-credential-socket\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.783307 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.783264    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-data\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.783615 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.783591    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istiod-ca-cert\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.785155 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.785135    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-envoy\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.785691 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.785670    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-podinfo\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.889262 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.881843    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-token\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.889262 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.882407    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mkhn\" (UniqueName: \"kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-kube-api-access-2mkhn\") pod \"e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") " pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:30.897974 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.897904    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:30.901760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:30.901736    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:31.337079 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:42:31.337051    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode017eb31_06e0_4b5d_8414_3df2bd85ecab.slice/crio-f6dde50937aa7b1cb7fc3cdf8cf6598560be16212eb727b196e127ba5dfea67d WatchSource:0}: Error finding container f6dde50937aa7b1cb7fc3cdf8cf6598560be16212eb727b196e127ba5dfea67d: Status 404 returned error can't find the container with id f6dde50937aa7b1cb7fc3cdf8cf6598560be16212eb727b196e127ba5dfea67d
Oct 31 18:42:31.337447 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:31.337207    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"]
Oct 31 18:42:31.339210 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:31.339178    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:42:31.339280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:31.339248    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:42:31.339319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:31.339280    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:42:31.723991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:31.723895    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" event={"ID":"e017eb31-06e0-4b5d-8414-3df2bd85ecab","Type":"ContainerStarted","Data":"c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a"}
Oct 31 18:42:31.723991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:31.723951    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" event={"ID":"e017eb31-06e0-4b5d-8414-3df2bd85ecab","Type":"ContainerStarted","Data":"f6dde50937aa7b1cb7fc3cdf8cf6598560be16212eb727b196e127ba5dfea67d"}
Oct 31 18:42:31.786156 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:31.786050    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:31.843550 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:31.843462    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" podStartSLOduration=1.843447151 podStartE2EDuration="1.843447151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:42:31.842802107 +0000 UTC m=+1297.989251970" watchObservedRunningTime="2026-10-31 18:42:31.843447151 +0000 UTC m=+1297.989897015"
Oct 31 18:42:31.901937 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:31.901902    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:32.907834 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:32.907806    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:33.773499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:33.767710    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:33.800758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:33.799722    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:42:34.814101 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:34.814059    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m" podUID="022ec6a8-e173-4976-b944-bcaef951597a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:42:34.872125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:34.872090    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:34.927178 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:34.926598    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv" podUID="3b6b07cc-6154-4582-8e24-fc085949657a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:42:35.045681 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:35.045320    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:35.367755 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:35.367720    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:35.468696 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:35.468662    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"]
Oct 31 18:42:35.473556 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:35.473522    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m" podUID="022ec6a8-e173-4976-b944-bcaef951597a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:42:35.544306 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:35.544267    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:35.757601 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:35.733062    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:38.131620 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:38.131584    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:40.471379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.471321    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m" podUID="022ec6a8-e173-4976-b944-bcaef951597a" containerName="payload-processing" containerID="cri-o://61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb" gracePeriod=25
Oct 31 18:42:40.741975 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.741950    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:40.868173 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.868142    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-kube-api-access-zm4ps\" (UniqueName: \"kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-kube-api-access-zm4ps\") pod \"022ec6a8-e173-4976-b944-bcaef951597a\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") "
Oct 31 18:42:40.868360 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.868186    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-plugins-config-volume\") pod \"022ec6a8-e173-4976-b944-bcaef951597a\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") "
Oct 31 18:42:40.868360 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.868295    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-routing-overlay-volume\") pod \"022ec6a8-e173-4976-b944-bcaef951597a\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") "
Oct 31 18:42:40.868360 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.868345    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-provider-credentials\") pod \"022ec6a8-e173-4976-b944-bcaef951597a\" (UID: \"022ec6a8-e173-4976-b944-bcaef951597a\") "
Oct 31 18:42:40.868703 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.868677    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-plugins-config-volume" pod "022ec6a8-e173-4976-b944-bcaef951597a" (UID: "022ec6a8-e173-4976-b944-bcaef951597a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:42:40.868703 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.868685    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-routing-overlay-volume" pod "022ec6a8-e173-4976-b944-bcaef951597a" (UID: "022ec6a8-e173-4976-b944-bcaef951597a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:42:40.868883 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.868864    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:42:40.868883 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.868884    4484 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/022ec6a8-e173-4976-b944-bcaef951597a-routing-overlay-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:42:40.871131 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.871104    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-kube-api-access-zm4ps" pod "022ec6a8-e173-4976-b944-bcaef951597a" (UID: "022ec6a8-e173-4976-b944-bcaef951597a"). InnerVolumeSpecName "kube-api-access-zm4ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:42:40.871131 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.871116    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-provider-credentials" pod "022ec6a8-e173-4976-b944-bcaef951597a" (UID: "022ec6a8-e173-4976-b944-bcaef951597a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:42:40.969771 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.969741    4484 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-provider-credentials\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:42:40.969771 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.969767    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm4ps\" (UniqueName: \"kubernetes.io/projected/022ec6a8-e173-4976-b944-bcaef951597a-kube-api-access-zm4ps\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:42:40.971911 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.971841    4484 generic.go:356] "Generic (PLEG): container finished" podID="022ec6a8-e173-4976-b944-bcaef951597a" containerID="61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb" exitCode=0
Oct 31 18:42:40.971911 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.971890    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"
Oct 31 18:42:40.972051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.971926    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m" event={"ID":"022ec6a8-e173-4976-b944-bcaef951597a","Type":"ContainerDied","Data":"61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb"}
Oct 31 18:42:40.972051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.971980    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m" event={"ID":"022ec6a8-e173-4976-b944-bcaef951597a","Type":"ContainerDied","Data":"4270578dcadb1c9c830788e2f08f32a8ed2e57ae71dac84bee194e98501c60d6"}
Oct 31 18:42:40.972051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.972001    4484 scope.go:122] "RemoveContainer" containerID="61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb"
Oct 31 18:42:40.982630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.982610    4484 scope.go:122] "RemoveContainer" containerID="61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb"
Oct 31 18:42:40.982932 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:42:40.982913    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb\": container with ID starting with 61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb not found: ID does not exist" containerID="61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb"
Oct 31 18:42:40.982992 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:40.982942    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb"} err="failed to get container status \"61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb\": rpc error: code = NotFound desc = could not find container \"61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb\": container with ID starting with 61c31464926281380e9040ba75cdb586d064caad8615cd56c171d06172bd0feb not found: ID does not exist"
Oct 31 18:42:41.140092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:41.140055    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"]
Oct 31 18:42:41.143351 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:41.143323    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f55c59c9-p6k8m"]
Oct 31 18:42:41.974780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:41.974744    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:41.993933 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:41.993908    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="022ec6a8-e173-4976-b944-bcaef951597a" path="/var/lib/kubelet/pods/022ec6a8-e173-4976-b944-bcaef951597a/volumes"
Oct 31 18:42:42.171204 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:42.171165    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:42.967997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:42.967969    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:47.447090 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:47.447058    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:47.652832 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:47.652801    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:52.082395 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:52.082357    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:52.281784 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:52.281748    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:54.264630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:54.264599    4484 scope.go:122] "RemoveContainer" containerID="5ae53c5f88c841f4176864534cf08c6144c6500cab86897489a2c2bee3d4cbec"
Oct 31 18:42:54.347201 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:54.347168    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:42:55.468893 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:42:55.468858    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:42:55.800899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:55.800865    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:42:57.065155 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:42:57.065123    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:00.496325 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:00.496288    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:00.837067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:00.836979    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:02.475105 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:02.475074    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:06.375293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.375263    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"]
Oct 31 18:43:06.377660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.375897    4484 cpu_manager.go:414] "containerMap: removing container" podUID="022ec6a8-e173-4976-b944-bcaef951597a" containerName="payload-processing"
Oct 31 18:43:06.377660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.375909    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="022ec6a8-e173-4976-b944-bcaef951597a" containerName="payload-processing"
Oct 31 18:43:06.377660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.376003    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="022ec6a8-e173-4976-b944-bcaef951597a" containerName="payload-processing"
Oct 31 18:43:06.379250 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.379231    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:06.437806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.437784    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"]
Oct 31 18:43:06.482736 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.482708    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:43:06.482957 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.482938    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:43:06.540829 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.540804    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64e37a8f-7lwmw\" (UID: \"d879555f-653a-47b5-8d70-bf28df4242b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:06.540974 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.540847    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtxxz\" (UniqueName: \"kubernetes.io/projected/d879555f-653a-47b5-8d70-bf28df4242b7-kube-api-access-dtxxz\") pod \"maas-api-revoke-sub-64e37a8f-7lwmw\" (UID: \"d879555f-653a-47b5-8d70-bf28df4242b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:06.541068 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.541047    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64e37a8f-7lwmw\" (UID: \"d879555f-653a-47b5-8d70-bf28df4242b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:06.641997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.641916    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64e37a8f-7lwmw\" (UID: \"d879555f-653a-47b5-8d70-bf28df4242b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:06.642149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.641996    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64e37a8f-7lwmw\" (UID: \"d879555f-653a-47b5-8d70-bf28df4242b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:06.642149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.642031    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtxxz\" (UniqueName: \"kubernetes.io/projected/d879555f-653a-47b5-8d70-bf28df4242b7-kube-api-access-dtxxz\") pod \"maas-api-revoke-sub-64e37a8f-7lwmw\" (UID: \"d879555f-653a-47b5-8d70-bf28df4242b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:06.642683 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.642657    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64e37a8f-7lwmw\" (UID: \"d879555f-653a-47b5-8d70-bf28df4242b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:06.644337 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.644320    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64e37a8f-7lwmw\" (UID: \"d879555f-653a-47b5-8d70-bf28df4242b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:06.699828 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.699806    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtxxz\" (UniqueName: \"kubernetes.io/projected/d879555f-653a-47b5-8d70-bf28df4242b7-kube-api-access-dtxxz\") pod \"maas-api-revoke-sub-64e37a8f-7lwmw\" (UID: \"d879555f-653a-47b5-8d70-bf28df4242b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:06.992357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:06.992328    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:07.154520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:07.154467    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:07.366539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:07.366458    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"]
Oct 31 18:43:07.367019 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:43:07.366995    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd879555f_653a_47b5_8d70_bf28df4242b7.slice/crio-54132d310516ce2f44952c41bdf823cd67390e78e6bb2af5f7bc725ff72bbd16 WatchSource:0}: Error finding container 54132d310516ce2f44952c41bdf823cd67390e78e6bb2af5f7bc725ff72bbd16: Status 404 returned error can't find the container with id 54132d310516ce2f44952c41bdf823cd67390e78e6bb2af5f7bc725ff72bbd16
Oct 31 18:43:08.200123 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:08.200084    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw" event={"ID":"d879555f-653a-47b5-8d70-bf28df4242b7","Type":"ContainerStarted","Data":"4ff350d3ae5b345ad5b1583b1ac68137b5fc9eb86c26c7ac5d4642a0089aac4b"}
Oct 31 18:43:08.200606 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:08.200139    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw" event={"ID":"d879555f-653a-47b5-8d70-bf28df4242b7","Type":"ContainerStarted","Data":"54132d310516ce2f44952c41bdf823cd67390e78e6bb2af5f7bc725ff72bbd16"}
Oct 31 18:43:08.314462 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:08.314403    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw" podStartSLOduration=2.314387567 podStartE2EDuration="2.314387567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:43:08.313756712 +0000 UTC m=+1334.460206575" watchObservedRunningTime="2026-10-31 18:43:08.314387567 +0000 UTC m=+1334.460837431"
Oct 31 18:43:09.207760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:09.207730    4484 generic.go:356] "Generic (PLEG): container finished" podID="d879555f-653a-47b5-8d70-bf28df4242b7" containerID="4ff350d3ae5b345ad5b1583b1ac68137b5fc9eb86c26c7ac5d4642a0089aac4b" exitCode=0
Oct 31 18:43:09.208165 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:09.207818    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw" event={"ID":"d879555f-653a-47b5-8d70-bf28df4242b7","Type":"ContainerDied","Data":"4ff350d3ae5b345ad5b1583b1ac68137b5fc9eb86c26c7ac5d4642a0089aac4b"}
Oct 31 18:43:10.369113 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:10.369088    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:10.395940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:10.395912    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d879555f-653a-47b5-8d70-bf28df4242b7-kube-api-access-dtxxz\" (UniqueName: \"kubernetes.io/projected/d879555f-653a-47b5-8d70-bf28df4242b7-kube-api-access-dtxxz\") pod \"d879555f-653a-47b5-8d70-bf28df4242b7\" (UID: \"d879555f-653a-47b5-8d70-bf28df4242b7\") "
Oct 31 18:43:10.396078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:10.395949    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-openshift-ca\") pod \"d879555f-653a-47b5-8d70-bf28df4242b7\" (UID: \"d879555f-653a-47b5-8d70-bf28df4242b7\") "
Oct 31 18:43:10.396078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:10.395966    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-xks-ca\") pod \"d879555f-653a-47b5-8d70-bf28df4242b7\" (UID: \"d879555f-653a-47b5-8d70-bf28df4242b7\") "
Oct 31 18:43:10.396289 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:10.396270    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-openshift-ca" pod "d879555f-653a-47b5-8d70-bf28df4242b7" (UID: "d879555f-653a-47b5-8d70-bf28df4242b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:43:10.396462 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:10.396449    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:43:10.398185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:10.398158    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-xks-ca" pod "d879555f-653a-47b5-8d70-bf28df4242b7" (UID: "d879555f-653a-47b5-8d70-bf28df4242b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:43:10.398288 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:10.398242    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d879555f-653a-47b5-8d70-bf28df4242b7-kube-api-access-dtxxz" pod "d879555f-653a-47b5-8d70-bf28df4242b7" (UID: "d879555f-653a-47b5-8d70-bf28df4242b7"). InnerVolumeSpecName "kube-api-access-dtxxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:43:10.498228 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:10.498200    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtxxz\" (UniqueName: \"kubernetes.io/projected/d879555f-653a-47b5-8d70-bf28df4242b7-kube-api-access-dtxxz\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:43:10.498228 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:10.498228    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d879555f-653a-47b5-8d70-bf28df4242b7-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:43:11.221330 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.221301    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"
Oct 31 18:43:11.221526 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.221329    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw" event={"ID":"d879555f-653a-47b5-8d70-bf28df4242b7","Type":"ContainerDied","Data":"54132d310516ce2f44952c41bdf823cd67390e78e6bb2af5f7bc725ff72bbd16"}
Oct 31 18:43:11.221526 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.221369    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54132d310516ce2f44952c41bdf823cd67390e78e6bb2af5f7bc725ff72bbd16"
Oct 31 18:43:11.496682 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.496656    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"]
Oct 31 18:43:11.497253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.497240    4484 cpu_manager.go:414] "containerMap: removing container" podUID="d879555f-653a-47b5-8d70-bf28df4242b7" containerName="revoke-keys"
Oct 31 18:43:11.497303 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.497254    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d879555f-653a-47b5-8d70-bf28df4242b7" containerName="revoke-keys"
Oct 31 18:43:11.497381 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.497371    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d879555f-653a-47b5-8d70-bf28df4242b7" containerName="revoke-keys"
Oct 31 18:43:11.500894 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.500877    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:11.510923 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.510898    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-metrics-certs\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:11.511049 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.510942    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cm6p\" (UniqueName: \"kubernetes.io/projected/5f80435b-76fb-41c8-a6a6-c5d99fe23021-kube-api-access-6cm6p\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:11.511156 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.511139    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-maas-api-tls\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:11.511205 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.511165    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f80435b-76fb-41c8-a6a6-c5d99fe23021-openshift-service-ca\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:11.554096 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.554065    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bd6c0a-serving-cert\""
Oct 31 18:43:11.585986 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.585958    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"]
Oct 31 18:43:11.613098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.613063    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-maas-api-tls\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:11.613275 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.613106    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f80435b-76fb-41c8-a6a6-c5d99fe23021-openshift-service-ca\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:11.613275 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:43:11.613205    4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-serving-cert: secret "maas-api-e2e-api-a-bd6c0a-serving-cert" not found
Oct 31 18:43:11.613275 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.613221    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-metrics-certs\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:11.613275 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:43:11.613263    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-maas-api-tls podName:5f80435b-76fb-41c8-a6a6-c5d99fe23021 nodeName:}" failed. No retries permitted until 2026-10-31 18:43:12.113248722 +0000 UTC m=+1338.259698579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-maas-api-tls") pod "maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp" (UID: "5f80435b-76fb-41c8-a6a6-c5d99fe23021") : secret "maas-api-e2e-api-a-bd6c0a-serving-cert" not found
Oct 31 18:43:11.613536 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.613341    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cm6p\" (UniqueName: \"kubernetes.io/projected/5f80435b-76fb-41c8-a6a6-c5d99fe23021-kube-api-access-6cm6p\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:11.614020 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.613996    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f80435b-76fb-41c8-a6a6-c5d99fe23021-openshift-service-ca\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:11.616292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.616273    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-metrics-certs\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:11.673595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.673570    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cm6p\" (UniqueName: \"kubernetes.io/projected/5f80435b-76fb-41c8-a6a6-c5d99fe23021-kube-api-access-6cm6p\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:11.707804 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.707774    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:11.907222 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:11.907126    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:12.119153 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:12.119121    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-maas-api-tls\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:12.121631 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:12.121602    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-maas-api-tls\") pod \"maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:12.414037 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:12.414000    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:12.789726 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:12.789698    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"]
Oct 31 18:43:12.790144 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:43:12.790035    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f80435b_76fb_41c8_a6a6_c5d99fe23021.slice/crio-5292009ebe9f425400bed5f09bf335c261da89a011595fe4916ccea5ab60e0c5 WatchSource:0}: Error finding container 5292009ebe9f425400bed5f09bf335c261da89a011595fe4916ccea5ab60e0c5: Status 404 returned error can't find the container with id 5292009ebe9f425400bed5f09bf335c261da89a011595fe4916ccea5ab60e0c5
Oct 31 18:43:12.791917 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:12.791898    4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:43:13.236401 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:13.236374    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp" event={"ID":"5f80435b-76fb-41c8-a6a6-c5d99fe23021","Type":"ContainerStarted","Data":"5292009ebe9f425400bed5f09bf335c261da89a011595fe4916ccea5ab60e0c5"}
Oct 31 18:43:14.245953 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:14.245914    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp" event={"ID":"5f80435b-76fb-41c8-a6a6-c5d99fe23021","Type":"ContainerStarted","Data":"5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76"}
Oct 31 18:43:14.246333 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:14.246032    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:14.363026 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:14.362972    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp" podStartSLOduration=2.526578671 podStartE2EDuration="3.362957663s" totalImagesPullingTime="836.378992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:43:14.362745834 +0000 UTC m=+1340.509195700" watchObservedRunningTime="2026-10-31 18:43:14.362957663 +0000 UTC m=+1340.509407558"
Oct 31 18:43:14.405744 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:14.405710    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:15.355020 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:15.354982    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:15.635863 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:15.635787    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:16.676870 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:16.676829    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:16.998713 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:16.998678    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:19.771196 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:19.771163    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:19.953858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:19.953821    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:20.131538 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:20.131434    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:20.258341 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:20.258312    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:43:21.797283 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:21.797231    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:22.880586 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:22.880557    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:23.107699 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.107637    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:23.257197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.257146    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:23.417576 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.417548    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"]
Oct 31 18:43:23.421896 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.421881    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"
Oct 31 18:43:23.479150 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.479113    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"]
Oct 31 18:43:23.487942 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.483597    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96f8b\" (UniqueName: \"kubernetes.io/projected/cdfc1774-5d73-4c87-8c82-141edcb3a067-kube-api-access-96f8b\") pod \"payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j\" (UID: \"cdfc1774-5d73-4c87-8c82-141edcb3a067\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"
Oct 31 18:43:23.487942 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.483667    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdfc1774-5d73-4c87-8c82-141edcb3a067-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j\" (UID: \"cdfc1774-5d73-4c87-8c82-141edcb3a067\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"
Oct 31 18:43:23.539603 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.539514    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a6c8da-dockercfg-cks9j\""
Oct 31 18:43:23.539908 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.539889    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a6c8da\""
Oct 31 18:43:23.572093 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.572062    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"]
Oct 31 18:43:23.583659 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.583567    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"
Oct 31 18:43:23.585728 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.585360    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96f8b\" (UniqueName: \"kubernetes.io/projected/cdfc1774-5d73-4c87-8c82-141edcb3a067-kube-api-access-96f8b\") pod \"payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j\" (UID: \"cdfc1774-5d73-4c87-8c82-141edcb3a067\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"
Oct 31 18:43:23.585728 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.585432    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdfc1774-5d73-4c87-8c82-141edcb3a067-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j\" (UID: \"cdfc1774-5d73-4c87-8c82-141edcb3a067\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"
Oct 31 18:43:23.586344 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.586319    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdfc1774-5d73-4c87-8c82-141edcb3a067-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j\" (UID: \"cdfc1774-5d73-4c87-8c82-141edcb3a067\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"
Oct 31 18:43:23.598028 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.597979    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"]
Oct 31 18:43:23.642838 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.642805    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96f8b\" (UniqueName: \"kubernetes.io/projected/cdfc1774-5d73-4c87-8c82-141edcb3a067-kube-api-access-96f8b\") pod \"payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j\" (UID: \"cdfc1774-5d73-4c87-8c82-141edcb3a067\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"
Oct 31 18:43:23.687300 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.687271    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtxf\" (UniqueName: \"kubernetes.io/projected/88244ad4-1418-4590-8e3d-f7ab7222482f-kube-api-access-cvtxf\") pod \"payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s\" (UID: \"88244ad4-1418-4590-8e3d-f7ab7222482f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"
Oct 31 18:43:23.687444 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.687316    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88244ad4-1418-4590-8e3d-f7ab7222482f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s\" (UID: \"88244ad4-1418-4590-8e3d-f7ab7222482f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"
Oct 31 18:43:23.735548 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.735524    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"
Oct 31 18:43:23.789415 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.789378    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtxf\" (UniqueName: \"kubernetes.io/projected/88244ad4-1418-4590-8e3d-f7ab7222482f-kube-api-access-cvtxf\") pod \"payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s\" (UID: \"88244ad4-1418-4590-8e3d-f7ab7222482f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"
Oct 31 18:43:23.789575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.789455    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88244ad4-1418-4590-8e3d-f7ab7222482f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s\" (UID: \"88244ad4-1418-4590-8e3d-f7ab7222482f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"
Oct 31 18:43:23.790344 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.790164    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88244ad4-1418-4590-8e3d-f7ab7222482f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s\" (UID: \"88244ad4-1418-4590-8e3d-f7ab7222482f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"
Oct 31 18:43:23.847026 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.846995    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtxf\" (UniqueName: \"kubernetes.io/projected/88244ad4-1418-4590-8e3d-f7ab7222482f-kube-api-access-cvtxf\") pod \"payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s\" (UID: \"88244ad4-1418-4590-8e3d-f7ab7222482f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"
Oct 31 18:43:23.908550 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:23.908516    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"
Oct 31 18:43:24.115195 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:24.115168    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"]
Oct 31 18:43:24.115416 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:43:24.115386    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfc1774_5d73_4c87_8c82_141edcb3a067.slice/crio-886ae47bf1dd8318bf9d8fd1c749b1a537e661fc2c08106fb73ef3c7256f7a66 WatchSource:0}: Error finding container 886ae47bf1dd8318bf9d8fd1c749b1a537e661fc2c08106fb73ef3c7256f7a66: Status 404 returned error can't find the container with id 886ae47bf1dd8318bf9d8fd1c749b1a537e661fc2c08106fb73ef3c7256f7a66
Oct 31 18:43:24.287384 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:43:24.287349    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88244ad4_1418_4590_8e3d_f7ab7222482f.slice/crio-d04e04474fda82f37d111fd5036c3bc3e44734fba5ec1a7cd0d2173206190cda WatchSource:0}: Error finding container d04e04474fda82f37d111fd5036c3bc3e44734fba5ec1a7cd0d2173206190cda: Status 404 returned error can't find the container with id d04e04474fda82f37d111fd5036c3bc3e44734fba5ec1a7cd0d2173206190cda
Oct 31 18:43:24.288019 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:24.287925    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"]
Oct 31 18:43:24.331139 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:24.331111    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j" event={"ID":"cdfc1774-5d73-4c87-8c82-141edcb3a067","Type":"ContainerStarted","Data":"8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8"}
Oct 31 18:43:24.331253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:24.331163    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j" event={"ID":"cdfc1774-5d73-4c87-8c82-141edcb3a067","Type":"ContainerStarted","Data":"886ae47bf1dd8318bf9d8fd1c749b1a537e661fc2c08106fb73ef3c7256f7a66"}
Oct 31 18:43:24.331253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:24.331186    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"
Oct 31 18:43:24.332426 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:24.332406    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s" event={"ID":"88244ad4-1418-4590-8e3d-f7ab7222482f","Type":"ContainerStarted","Data":"d04e04474fda82f37d111fd5036c3bc3e44734fba5ec1a7cd0d2173206190cda"}
Oct 31 18:43:24.446051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:24.445962    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j" podStartSLOduration=1.445949443 podStartE2EDuration="1.445949443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:43:24.444671641 +0000 UTC m=+1350.591121509" watchObservedRunningTime="2026-10-31 18:43:24.445949443 +0000 UTC m=+1350.592399301"
Oct 31 18:43:25.340909 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:25.340876    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s" event={"ID":"88244ad4-1418-4590-8e3d-f7ab7222482f","Type":"ContainerStarted","Data":"75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2"}
Oct 31 18:43:25.341370 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:25.340955    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"
Oct 31 18:43:25.458929 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:25.458873    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s" podStartSLOduration=2.458855466 podStartE2EDuration="2.458855466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:43:25.455380501 +0000 UTC m=+1351.601830366" watchObservedRunningTime="2026-10-31 18:43:25.458855466 +0000 UTC m=+1351.605305331"
Oct 31 18:43:25.471502 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:43:25.471456    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:43:26.595080 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:26.595041    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:26.964573 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:26.964463    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:27.117930 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:27.117898    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:31.259427 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:31.259391    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:31.363765 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:31.363713    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:43:35.344143 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:35.344119    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"
Oct 31 18:43:35.968498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:35.967942    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:36.351565 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:36.351532    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"
Oct 31 18:43:41.517837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:41.517803    4484 generic.go:356] "Generic (PLEG): container finished" podID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerID="c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a" exitCode=0
Oct 31 18:43:41.518319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:41.517875    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" event={"ID":"e017eb31-06e0-4b5d-8414-3df2bd85ecab","Type":"ContainerDied","Data":"c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a"}
Oct 31 18:43:41.518319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:41.518295    4484 scope.go:122] "RemoveContainer" containerID="c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a"
Oct 31 18:43:41.519705 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:41.519675    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:43:41.519788 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:41.519741    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:43:41.519788 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:41.519771    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:43:42.233304 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:42.233261    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:42.527100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:42.527065    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" event={"ID":"e017eb31-06e0-4b5d-8414-3df2bd85ecab","Type":"ContainerStarted","Data":"9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e"}
Oct 31 18:43:42.902929 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:42.902827    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:43:42.903110 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:42.903019    4484 patch_prober.go:29] interesting pod/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 18:43:42.903110 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:42.903083    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 18:43:43.902182 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:43.902145    4484 patch_prober.go:29] interesting pod/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 18:43:43.902718 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:43.902204    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 18:43:44.902969 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:44.902929    4484 patch_prober.go:29] interesting pod/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 18:43:44.903508 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:44.903013    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 18:43:45.336408 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.336372    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:45.341834 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.341802    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"]
Oct 31 18:43:45.347725 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.347703    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.411764 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.411729    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"]
Oct 31 18:43:45.423342 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.423309    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-model-cache\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.423629 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.423605    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-tmp-dir\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.423780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.423765    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e451c404-1138-48ef-928e-51aa8584f45d-tls-certs\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.423959 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.423938    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmgtj\" (UniqueName: \"kubernetes.io/projected/e451c404-1138-48ef-928e-51aa8584f45d-kube-api-access-pmgtj\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.424162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.424145    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-home\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.424527 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.424508    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-dshm\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.456488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.456445    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3417b822\"/\"kube-root-ca.crt\""
Oct 31 18:43:45.456618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.456513    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3417b822\"/\"openshift-service-ca.crt\""
Oct 31 18:43:45.456618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.456565    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3417b822\"/\"default-dockercfg-bwtr9\""
Oct 31 18:43:45.456618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.456607    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3417b822\"/\"e2e-lifecycle-model-3417b822-kserve-self-signed-certs\""
Oct 31 18:43:45.526199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.526168    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-dshm\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.526621 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.526598    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-model-cache\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.526764 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.526628    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-tmp-dir\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.526764 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.526647    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e451c404-1138-48ef-928e-51aa8584f45d-tls-certs\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.526764 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.526663    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmgtj\" (UniqueName: \"kubernetes.io/projected/e451c404-1138-48ef-928e-51aa8584f45d-kube-api-access-pmgtj\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.526764 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.526729    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-home\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.527151 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.527127    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-tmp-dir\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.527230 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.527132    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-home\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.527461 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.527439    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-model-cache\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.528890 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.528868    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:45.529200 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.529180    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-dshm\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.531999 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.531972    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e451c404-1138-48ef-928e-51aa8584f45d-tls-certs\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.596550 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.595638    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmgtj\" (UniqueName: \"kubernetes.io/projected/e451c404-1138-48ef-928e-51aa8584f45d-kube-api-access-pmgtj\") pod \"e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") " pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.685516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.682752    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:45.902653 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.902208    4484 patch_prober.go:29] interesting pod/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 18:43:45.902653 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:45.902255    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 18:43:46.088204 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:46.088175    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"]
Oct 31 18:43:46.088910 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:43:46.088884    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode451c404_1138_48ef_928e_51aa8584f45d.slice/crio-7a65110b2bb0c4a0d4b6940b935ccfae9ddeecb58b7e0ea799740804aa509ff2 WatchSource:0}: Error finding container 7a65110b2bb0c4a0d4b6940b935ccfae9ddeecb58b7e0ea799740804aa509ff2: Status 404 returned error can't find the container with id 7a65110b2bb0c4a0d4b6940b935ccfae9ddeecb58b7e0ea799740804aa509ff2
Oct 31 18:43:46.576522 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:46.576457    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc" event={"ID":"e451c404-1138-48ef-928e-51aa8584f45d","Type":"ContainerStarted","Data":"c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc"}
Oct 31 18:43:46.576759 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:46.576559    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc" event={"ID":"e451c404-1138-48ef-928e-51aa8584f45d","Type":"ContainerStarted","Data":"7a65110b2bb0c4a0d4b6940b935ccfae9ddeecb58b7e0ea799740804aa509ff2"}
Oct 31 18:43:46.689618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:46.689582    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:46.695339 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:46.695272    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc" podStartSLOduration=1.695257478 podStartE2EDuration="1.695257478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:43:46.69201469 +0000 UTC m=+1372.838464553" watchObservedRunningTime="2026-10-31 18:43:46.695257478 +0000 UTC m=+1372.841707341"
Oct 31 18:43:46.902805 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:46.902723    4484 patch_prober.go:29] interesting pod/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 31 18:43:46.902805 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:46.902782    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 31 18:43:47.771827 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:47.771790    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:47.907056 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:47.907029    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:43:47.907441 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:47.907418    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:43:47.908073 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:47.908052    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:43:50.045288 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:50.045248    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:52.417668 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.417629    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:52.464979 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.464949    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"]
Oct 31 18:43:52.469276 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.469258    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.522619 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.522592    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6279f5-openshift-default-dockercfg-d8cq6\""
Oct 31 18:43:52.552742 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.552705    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.552897 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.552819    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.552897 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.552860    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.553011 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.552992    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.553047 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.553026    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd95837-2a87-4361-877a-2d2edfd66977-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.553080 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.553047    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.553080 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.553064    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.553208 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.553197    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd95837-2a87-4361-877a-2d2edfd66977-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.553268 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.553254    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6tmh\" (UniqueName: \"kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-kube-api-access-k6tmh\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.577185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.577157    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"]
Oct 31 18:43:52.655116 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655086    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6tmh\" (UniqueName: \"kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-kube-api-access-k6tmh\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.655284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655146    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.655284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655208    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.655284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655245    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.655458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655292    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.655458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655322    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd95837-2a87-4361-877a-2d2edfd66977-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.655458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655419    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.655666 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655497    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.655666 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655603    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.655666 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655635    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd95837-2a87-4361-877a-2d2edfd66977-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.655853 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655831    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.655918 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655852    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.655918 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.655865    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.656177 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.656002    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd95837-2a87-4361-877a-2d2edfd66977-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.657637 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.657618    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.658036 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.658020    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd95837-2a87-4361-877a-2d2edfd66977-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.712100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.712036    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6tmh\" (UniqueName: \"kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-kube-api-access-k6tmh\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.712100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.712038    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:52.783703 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:52.783682    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:53.211136 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:43:53.211109    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd95837_2a87_4361_877a_2d2edfd66977.slice/crio-1e5827af6253ffbf0f148c7c19a9a9c302876a4cfbcfbc262449bb54de065e6a WatchSource:0}: Error finding container 1e5827af6253ffbf0f148c7c19a9a9c302876a4cfbcfbc262449bb54de065e6a: Status 404 returned error can't find the container with id 1e5827af6253ffbf0f148c7c19a9a9c302876a4cfbcfbc262449bb54de065e6a
Oct 31 18:43:53.211889 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:53.211866    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"]
Oct 31 18:43:53.213632 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:53.213600    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:43:53.213712 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:53.213678    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:43:53.213712 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:53.213708    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:43:53.636509 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:53.636456    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg" event={"ID":"bcd95837-2a87-4361-877a-2d2edfd66977","Type":"ContainerStarted","Data":"86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89"}
Oct 31 18:43:53.636938 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:53.636528    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg" event={"ID":"bcd95837-2a87-4361-877a-2d2edfd66977","Type":"ContainerStarted","Data":"1e5827af6253ffbf0f148c7c19a9a9c302876a4cfbcfbc262449bb54de065e6a"}
Oct 31 18:43:53.758430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:53.758373    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg" podStartSLOduration=1.758361028 podStartE2EDuration="1.758361028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:43:53.75750644 +0000 UTC m=+1379.903956304" watchObservedRunningTime="2026-10-31 18:43:53.758361028 +0000 UTC m=+1379.904810891"
Oct 31 18:43:53.786536 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:53.784002    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:43:54.787892 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:54.787860    4484 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 18:43:54.788537 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:54.787908    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg" podUID="bcd95837-2a87-4361-877a-2d2edfd66977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 18:43:54.858833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:54.858803    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:55.493143 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:43:55.493104    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:43:55.685500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:55.685443    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:55.696791 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:55.696767    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:55.785575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:55.784842    4484 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 18:43:55.785575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:55.784898    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg" podUID="bcd95837-2a87-4361-877a-2d2edfd66977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 18:43:55.918686 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:55.918649    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:56.094314 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:56.094174    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:56.694136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:56.694106    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:43:56.785605 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:56.785566    4484 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 18:43:56.785799 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:56.785629    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg" podUID="bcd95837-2a87-4361-877a-2d2edfd66977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 18:43:56.887042 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:56.887009    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:57.528915 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:57.528731    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:57.787908 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:57.787818    4484 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 18:43:57.788172 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:57.788143    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg" podUID="bcd95837-2a87-4361-877a-2d2edfd66977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 18:43:57.856263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:57.856234    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:43:58.785853 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:58.785817    4484 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 18:43:58.786278 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:58.785879    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg" podUID="bcd95837-2a87-4361-877a-2d2edfd66977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 18:43:58.887390 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:58.887342    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:43:59.920921 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:43:59.920883    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:00.092076 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:00.092042    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:00.201715 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:00.201633    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:44:00.202984 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:00.202956    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:44:00.203937 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:00.203914    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:44:00.256164 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:00.256129    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:02.238082 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:02.238048    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:02.452306 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:02.452271    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:05.104421 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:05.104386    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:05.409942 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:05.409858    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:05.585740 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:05.585699    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:05.959496 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:05.959435    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"]
Oct 31 18:44:05.968055 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:05.968020    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"
Oct 31 18:44:06.031535 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:06.030180    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"]
Oct 31 18:44:06.069823 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:06.069632    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-373109-dockercfg-9xcc8\""
Oct 31 18:44:06.071490 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:06.070183    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-373109\""
Oct 31 18:44:06.087365 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:06.086847    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d8865e-abfc-4a75-8b38-e5458b5eb570-plugins-config-volume\") pod \"payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl\" (UID: \"93d8865e-abfc-4a75-8b38-e5458b5eb570\") " pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"
Oct 31 18:44:06.093637 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:06.093594    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvz46\" (UniqueName: \"kubernetes.io/projected/93d8865e-abfc-4a75-8b38-e5458b5eb570-kube-api-access-bvz46\") pod \"payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl\" (UID: \"93d8865e-abfc-4a75-8b38-e5458b5eb570\") " pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"
Oct 31 18:44:06.195037 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:06.195008    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d8865e-abfc-4a75-8b38-e5458b5eb570-plugins-config-volume\") pod \"payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl\" (UID: \"93d8865e-abfc-4a75-8b38-e5458b5eb570\") " pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"
Oct 31 18:44:06.195436 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:06.195053    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvz46\" (UniqueName: \"kubernetes.io/projected/93d8865e-abfc-4a75-8b38-e5458b5eb570-kube-api-access-bvz46\") pod \"payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl\" (UID: \"93d8865e-abfc-4a75-8b38-e5458b5eb570\") " pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"
Oct 31 18:44:06.195707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:06.195687    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d8865e-abfc-4a75-8b38-e5458b5eb570-plugins-config-volume\") pod \"payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl\" (UID: \"93d8865e-abfc-4a75-8b38-e5458b5eb570\") " pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"
Oct 31 18:44:06.254369 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:06.254342    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvz46\" (UniqueName: \"kubernetes.io/projected/93d8865e-abfc-4a75-8b38-e5458b5eb570-kube-api-access-bvz46\") pod \"payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl\" (UID: \"93d8865e-abfc-4a75-8b38-e5458b5eb570\") " pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"
Oct 31 18:44:06.316195 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:06.316164    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"
Oct 31 18:44:06.691706 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:44:06.691672    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d8865e_abfc_4a75_8b38_e5458b5eb570.slice/crio-2f43d44013b18e2395aec1657354d5ca611e996d648bb4abda700feabfa6cdaf WatchSource:0}: Error finding container 2f43d44013b18e2395aec1657354d5ca611e996d648bb4abda700feabfa6cdaf: Status 404 returned error can't find the container with id 2f43d44013b18e2395aec1657354d5ca611e996d648bb4abda700feabfa6cdaf
Oct 31 18:44:06.691872 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:06.691732    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"]
Oct 31 18:44:06.762069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:06.762038    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl" event={"ID":"93d8865e-abfc-4a75-8b38-e5458b5eb570","Type":"ContainerStarted","Data":"2f43d44013b18e2395aec1657354d5ca611e996d648bb4abda700feabfa6cdaf"}
Oct 31 18:44:07.770392 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:07.770357    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl" event={"ID":"93d8865e-abfc-4a75-8b38-e5458b5eb570","Type":"ContainerStarted","Data":"9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4"}
Oct 31 18:44:07.770829 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:07.770436    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"
Oct 31 18:44:07.887595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:07.887545    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl" podStartSLOduration=2.887530201 podStartE2EDuration="2.887530201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:44:07.883698274 +0000 UTC m=+1394.030148139" watchObservedRunningTime="2026-10-31 18:44:07.887530201 +0000 UTC m=+1394.033980064"
Oct 31 18:44:10.074384 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.074346    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:10.195641 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.195609    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"]
Oct 31 18:44:10.199961 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.199939    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.260308 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.260277    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"]
Oct 31 18:44:10.304703 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.304673    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6279f5\"/\"openshift-service-ca.crt\""
Oct 31 18:44:10.304879 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.304716    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6279f5\"/\"default-dockercfg-hwl4n\""
Oct 31 18:44:10.304879 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.304759    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-6279f5\"/\"distinct-2-w4-6279f5-kserve-self-signed-certs\""
Oct 31 18:44:10.304879 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.304750    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-6279f5\"/\"kube-root-ca.crt\""
Oct 31 18:44:10.307374 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.307350    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:10.346802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.346610    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-home\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.347227 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.347183    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-dshm\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.347463 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.347385    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-model-cache\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.348109 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.347810    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fmqv\" (UniqueName: \"kubernetes.io/projected/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-kube-api-access-8fmqv\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.348109 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.347893    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tmp-dir\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.348109 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.347930    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tls-certs\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.449264 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.449234    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-dshm\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.449459 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.449267    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-model-cache\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.449459 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.449410    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fmqv\" (UniqueName: \"kubernetes.io/projected/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-kube-api-access-8fmqv\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.449664 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.449506    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tmp-dir\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.449664 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.449538    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tls-certs\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.449664 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.449618    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-home\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.449851 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.449830    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-model-cache\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.449916 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.449885    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tmp-dir\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.449958 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.449939    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-home\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.451695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.451676    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-dshm\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.452073 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.452057    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tls-certs\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.506909 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.506875    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fmqv\" (UniqueName: \"kubernetes.io/projected/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-kube-api-access-8fmqv\") pod \"distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") " pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.512761 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.512737    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:10.919911 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:44:10.919881    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ae7a3ad_06bd_40ff_a5c0_9aa7410a57ab.slice/crio-08d87a495bbe36aeca6a4ccd252cb652edf86ca0e983152390e53582f990244f WatchSource:0}: Error finding container 08d87a495bbe36aeca6a4ccd252cb652edf86ca0e983152390e53582f990244f: Status 404 returned error can't find the container with id 08d87a495bbe36aeca6a4ccd252cb652edf86ca0e983152390e53582f990244f
Oct 31 18:44:10.920062 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:10.919957    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"]
Oct 31 18:44:11.797694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:11.797662    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k" event={"ID":"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab","Type":"ContainerStarted","Data":"b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d"}
Oct 31 18:44:11.798085 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:11.797708    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k" event={"ID":"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab","Type":"ContainerStarted","Data":"08d87a495bbe36aeca6a4ccd252cb652edf86ca0e983152390e53582f990244f"}
Oct 31 18:44:11.913967 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:11.913910    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k" podStartSLOduration=1.913896978 podStartE2EDuration="1.913896978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:44:11.91188415 +0000 UTC m=+1398.058334016" watchObservedRunningTime="2026-10-31 18:44:11.913896978 +0000 UTC m=+1398.060346884"
Oct 31 18:44:12.513646 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:12.513612    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:14.763280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:14.763245    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:16.557431 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:16.557402    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:16.712753 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:16.712719    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:18.779794 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:18.779767    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"
Oct 31 18:44:19.555737 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:19.555701    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:20.513923 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:20.513886    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:20.526493 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:20.526442    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:20.868601 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:20.868521    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 18:44:24.360712 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:24.360680    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:25.469144 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:44:25.469106    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:44:29.082198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:29.082161    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:33.857199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:33.857162    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:38.490332 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:38.490294    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:39.061885 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:39.061853    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:44:43.046780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:43.046745    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"]
Oct 31 18:44:43.097492 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:43.097450    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a990b93f-5n52h"]
Oct 31 18:44:43.265101 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:43.265066    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:43.996686 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:43.996655    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705003bc-7509-4637-88c7-267931dfcc77" path="/var/lib/kubelet/pods/705003bc-7509-4637-88c7-267931dfcc77/volumes"
Oct 31 18:44:47.482284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:47.482243    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:47.789617 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:47.789582    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:47.977042 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:47.977004    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:48.134860 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:48.134757    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:48.301841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:48.301806    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:51.432736 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:51.432700    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:52.156694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:52.156651    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:52.763406 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:52.763360    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:52.926265 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:52.926227    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:53.093679 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:53.093591    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:53.242294 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:53.242264    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:55.468305 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:44:55.468263    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:44:57.046977 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:57.046944    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"]
Oct 31 18:44:57.097115 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:57.097091    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4387450c-ld8xt"]
Oct 31 18:44:57.637179 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:57.637138    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:57.895960 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:57.895873    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:57.994516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:57.994462    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e096445e-efea-4b11-b690-f9fcbf00b109" path="/var/lib/kubelet/pods/e096445e-efea-4b11-b690-f9fcbf00b109/volumes"
Oct 31 18:44:58.139925 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:58.139895    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:58.317889 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:58.317862    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:58.487135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:58.487099    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:44:58.727368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:44:58.727271    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:00.158136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.158104    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"]
Oct 31 18:45:00.164443 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.164422    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"
Oct 31 18:45:00.166728 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.166708    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"]
Oct 31 18:45:00.170651 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.170631    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"
Oct 31 18:45:00.186074 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.186051    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"]
Oct 31 18:45:00.190864 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.190849    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"
Oct 31 18:45:00.208709 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.208685    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"]
Oct 31 18:45:00.212672 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.212657    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"
Oct 31 18:45:00.219938 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.219919    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 18:45:00.240776 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.240750    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bf7c50-4e6c-4787-8b53-79b480ad095f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l\" (UID: \"83bf7c50-4e6c-4787-8b53-79b480ad095f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"
Oct 31 18:45:00.240893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.240823    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26d5481-5858-4a2d-8bdc-067e04d73052-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp\" (UID: \"d26d5481-5858-4a2d-8bdc-067e04d73052\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"
Oct 31 18:45:00.240953 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.240925    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r96jc\" (UniqueName: \"kubernetes.io/projected/21cc53ff-5b7d-4c58-8bd2-aacecde51029-kube-api-access-r96jc\") pod \"maas-api-key-cleanup-29891205-q8g4d\" (UID: \"21cc53ff-5b7d-4c58-8bd2-aacecde51029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"
Oct 31 18:45:00.240999 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.240968    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfrvh\" (UniqueName: \"kubernetes.io/projected/d26d5481-5858-4a2d-8bdc-067e04d73052-kube-api-access-kfrvh\") pod \"maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp\" (UID: \"d26d5481-5858-4a2d-8bdc-067e04d73052\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"
Oct 31 18:45:00.241080 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.241064    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdrqt\" (UniqueName: \"kubernetes.io/projected/83bf7c50-4e6c-4787-8b53-79b480ad095f-kube-api-access-xdrqt\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l\" (UID: \"83bf7c50-4e6c-4787-8b53-79b480ad095f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"
Oct 31 18:45:00.241171 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.241157    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9c46c9-744f-48fd-85ac-43415d1f561c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb\" (UID: \"ce9c46c9-744f-48fd-85ac-43415d1f561c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"
Oct 31 18:45:00.241206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.241186    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9sqw\" (UniqueName: \"kubernetes.io/projected/ce9c46c9-744f-48fd-85ac-43415d1f561c-kube-api-access-n9sqw\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb\" (UID: \"ce9c46c9-744f-48fd-85ac-43415d1f561c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"
Oct 31 18:45:00.241344 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.241330    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21cc53ff-5b7d-4c58-8bd2-aacecde51029-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891205-q8g4d\" (UID: \"21cc53ff-5b7d-4c58-8bd2-aacecde51029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"
Oct 31 18:45:00.263248 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.263224    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"]
Oct 31 18:45:00.321909 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.321840    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"]
Oct 31 18:45:00.323568 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.323541    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"]
Oct 31 18:45:00.329017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.328991    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"]
Oct 31 18:45:00.342886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.342855    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9sqw\" (UniqueName: \"kubernetes.io/projected/ce9c46c9-744f-48fd-85ac-43415d1f561c-kube-api-access-n9sqw\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb\" (UID: \"ce9c46c9-744f-48fd-85ac-43415d1f561c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"
Oct 31 18:45:00.343050 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.343031    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21cc53ff-5b7d-4c58-8bd2-aacecde51029-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891205-q8g4d\" (UID: \"21cc53ff-5b7d-4c58-8bd2-aacecde51029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"
Oct 31 18:45:00.343126 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.343110    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bf7c50-4e6c-4787-8b53-79b480ad095f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l\" (UID: \"83bf7c50-4e6c-4787-8b53-79b480ad095f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"
Oct 31 18:45:00.343221 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.343205    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26d5481-5858-4a2d-8bdc-067e04d73052-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp\" (UID: \"d26d5481-5858-4a2d-8bdc-067e04d73052\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"
Oct 31 18:45:00.343268 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.343254    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96jc\" (UniqueName: \"kubernetes.io/projected/21cc53ff-5b7d-4c58-8bd2-aacecde51029-kube-api-access-r96jc\") pod \"maas-api-key-cleanup-29891205-q8g4d\" (UID: \"21cc53ff-5b7d-4c58-8bd2-aacecde51029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"
Oct 31 18:45:00.343303 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.343287    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfrvh\" (UniqueName: \"kubernetes.io/projected/d26d5481-5858-4a2d-8bdc-067e04d73052-kube-api-access-kfrvh\") pod \"maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp\" (UID: \"d26d5481-5858-4a2d-8bdc-067e04d73052\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"
Oct 31 18:45:00.343335 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.343316    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdrqt\" (UniqueName: \"kubernetes.io/projected/83bf7c50-4e6c-4787-8b53-79b480ad095f-kube-api-access-xdrqt\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l\" (UID: \"83bf7c50-4e6c-4787-8b53-79b480ad095f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"
Oct 31 18:45:00.343410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.343395    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9c46c9-744f-48fd-85ac-43415d1f561c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb\" (UID: \"ce9c46c9-744f-48fd-85ac-43415d1f561c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"
Oct 31 18:45:00.343990 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.343966    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21cc53ff-5b7d-4c58-8bd2-aacecde51029-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891205-q8g4d\" (UID: \"21cc53ff-5b7d-4c58-8bd2-aacecde51029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"
Oct 31 18:45:00.344221 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.344202    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9c46c9-744f-48fd-85ac-43415d1f561c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb\" (UID: \"ce9c46c9-744f-48fd-85ac-43415d1f561c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"
Oct 31 18:45:00.344937 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.344909    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bf7c50-4e6c-4787-8b53-79b480ad095f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l\" (UID: \"83bf7c50-4e6c-4787-8b53-79b480ad095f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"
Oct 31 18:45:00.345072 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.344933    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26d5481-5858-4a2d-8bdc-067e04d73052-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp\" (UID: \"d26d5481-5858-4a2d-8bdc-067e04d73052\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"
Oct 31 18:45:00.400537 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.400502    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9sqw\" (UniqueName: \"kubernetes.io/projected/ce9c46c9-744f-48fd-85ac-43415d1f561c-kube-api-access-n9sqw\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb\" (UID: \"ce9c46c9-744f-48fd-85ac-43415d1f561c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"
Oct 31 18:45:00.401133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.401104    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfrvh\" (UniqueName: \"kubernetes.io/projected/d26d5481-5858-4a2d-8bdc-067e04d73052-kube-api-access-kfrvh\") pod \"maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp\" (UID: \"d26d5481-5858-4a2d-8bdc-067e04d73052\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"
Oct 31 18:45:00.401369 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.401350    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r96jc\" (UniqueName: \"kubernetes.io/projected/21cc53ff-5b7d-4c58-8bd2-aacecde51029-kube-api-access-r96jc\") pod \"maas-api-key-cleanup-29891205-q8g4d\" (UID: \"21cc53ff-5b7d-4c58-8bd2-aacecde51029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"
Oct 31 18:45:00.401851 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.401830    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdrqt\" (UniqueName: \"kubernetes.io/projected/83bf7c50-4e6c-4787-8b53-79b480ad095f-kube-api-access-xdrqt\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l\" (UID: \"83bf7c50-4e6c-4787-8b53-79b480ad095f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"
Oct 31 18:45:00.481812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.481736    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"
Oct 31 18:45:00.487296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.487275    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"
Oct 31 18:45:00.504226 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.504203    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"
Oct 31 18:45:00.525857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.525818    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"
Oct 31 18:45:00.882397 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:45:00.882367    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9c46c9_744f_48fd_85ac_43415d1f561c.slice/crio-b8e4c7559e4e6536a0674a92d61a47b36efba3d0f180b265adb2cdf0c270dd5b WatchSource:0}: Error finding container b8e4c7559e4e6536a0674a92d61a47b36efba3d0f180b265adb2cdf0c270dd5b: Status 404 returned error can't find the container with id b8e4c7559e4e6536a0674a92d61a47b36efba3d0f180b265adb2cdf0c270dd5b
Oct 31 18:45:00.883063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.883043    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"]
Oct 31 18:45:00.929901 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:45:00.929869    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bf7c50_4e6c_4787_8b53_79b480ad095f.slice/crio-45fb3bfbb9c174f66c791568bcf51d553462a65317c12d039772b6e4f9ab6ffd WatchSource:0}: Error finding container 45fb3bfbb9c174f66c791568bcf51d553462a65317c12d039772b6e4f9ab6ffd: Status 404 returned error can't find the container with id 45fb3bfbb9c174f66c791568bcf51d553462a65317c12d039772b6e4f9ab6ffd
Oct 31 18:45:00.930279 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.930255    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"]
Oct 31 18:45:00.932122 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:45:00.932101    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26d5481_5858_4a2d_8bdc_067e04d73052.slice/crio-ef56e019b56409160ea8f532e03a2419cad525fb529edf0007d78a2d8b437f6f WatchSource:0}: Error finding container ef56e019b56409160ea8f532e03a2419cad525fb529edf0007d78a2d8b437f6f: Status 404 returned error can't find the container with id ef56e019b56409160ea8f532e03a2419cad525fb529edf0007d78a2d8b437f6f
Oct 31 18:45:00.935012 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.934993    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"]
Oct 31 18:45:00.981352 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:45:00.981325    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21cc53ff_5b7d_4c58_8bd2_aacecde51029.slice/crio-9ef388e99862db5b407562ab38cb953ea918fc2acfbe3eb83209165cb6bed9f2 WatchSource:0}: Error finding container 9ef388e99862db5b407562ab38cb953ea918fc2acfbe3eb83209165cb6bed9f2: Status 404 returned error can't find the container with id 9ef388e99862db5b407562ab38cb953ea918fc2acfbe3eb83209165cb6bed9f2
Oct 31 18:45:00.981794 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:00.981775    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"]
Oct 31 18:45:01.129237 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:01.129153    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d" event={"ID":"21cc53ff-5b7d-4c58-8bd2-aacecde51029","Type":"ContainerStarted","Data":"9ef388e99862db5b407562ab38cb953ea918fc2acfbe3eb83209165cb6bed9f2"}
Oct 31 18:45:01.130454 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:01.130430    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp" event={"ID":"d26d5481-5858-4a2d-8bdc-067e04d73052","Type":"ContainerStarted","Data":"ef56e019b56409160ea8f532e03a2419cad525fb529edf0007d78a2d8b437f6f"}
Oct 31 18:45:01.131673 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:01.131653    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb" event={"ID":"ce9c46c9-744f-48fd-85ac-43415d1f561c","Type":"ContainerStarted","Data":"b8e4c7559e4e6536a0674a92d61a47b36efba3d0f180b265adb2cdf0c270dd5b"}
Oct 31 18:45:01.132834 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:01.132818    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l" event={"ID":"83bf7c50-4e6c-4787-8b53-79b480ad095f","Type":"ContainerStarted","Data":"45fb3bfbb9c174f66c791568bcf51d553462a65317c12d039772b6e4f9ab6ffd"}
Oct 31 18:45:02.142333 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.142253    4484 generic.go:356] "Generic (PLEG): container finished" podID="ce9c46c9-744f-48fd-85ac-43415d1f561c" containerID="42158d348125118d2aaec562847788a97fb4763b13927e5e3ebdaf19b247fb84" exitCode=60
Oct 31 18:45:02.142777 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.142341    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb" event={"ID":"ce9c46c9-744f-48fd-85ac-43415d1f561c","Type":"ContainerDied","Data":"42158d348125118d2aaec562847788a97fb4763b13927e5e3ebdaf19b247fb84"}
Oct 31 18:45:02.142777 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.142628    4484 scope.go:122] "RemoveContainer" containerID="42158d348125118d2aaec562847788a97fb4763b13927e5e3ebdaf19b247fb84"
Oct 31 18:45:02.144113 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.144088    4484 generic.go:356] "Generic (PLEG): container finished" podID="83bf7c50-4e6c-4787-8b53-79b480ad095f" containerID="b6dfe7f36e1e64fc7e31ffc760ba2b61d82535539be8f2b9f84fee22bda6840b" exitCode=60
Oct 31 18:45:02.144226 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.144183    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l" event={"ID":"83bf7c50-4e6c-4787-8b53-79b480ad095f","Type":"ContainerDied","Data":"b6dfe7f36e1e64fc7e31ffc760ba2b61d82535539be8f2b9f84fee22bda6840b"}
Oct 31 18:45:02.144434 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.144418    4484 scope.go:122] "RemoveContainer" containerID="b6dfe7f36e1e64fc7e31ffc760ba2b61d82535539be8f2b9f84fee22bda6840b"
Oct 31 18:45:02.146133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.146115    4484 generic.go:356] "Generic (PLEG): container finished" podID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" containerID="3ee6510509621dbb0a9a5dae32679e1bdd3f87ccdb4c2ecc7126877958eb61da" exitCode=60
Oct 31 18:45:02.146309 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.146210    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d" event={"ID":"21cc53ff-5b7d-4c58-8bd2-aacecde51029","Type":"ContainerDied","Data":"3ee6510509621dbb0a9a5dae32679e1bdd3f87ccdb4c2ecc7126877958eb61da"}
Oct 31 18:45:02.146423 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.146407    4484 scope.go:122] "RemoveContainer" containerID="3ee6510509621dbb0a9a5dae32679e1bdd3f87ccdb4c2ecc7126877958eb61da"
Oct 31 18:45:02.148283 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.148263    4484 generic.go:356] "Generic (PLEG): container finished" podID="d26d5481-5858-4a2d-8bdc-067e04d73052" containerID="d38d25e9a859776ca585897af7ee6a02ab4c8f2d18481d7a96a17020ae2761ed" exitCode=60
Oct 31 18:45:02.148392 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.148331    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp" event={"ID":"d26d5481-5858-4a2d-8bdc-067e04d73052","Type":"ContainerDied","Data":"d38d25e9a859776ca585897af7ee6a02ab4c8f2d18481d7a96a17020ae2761ed"}
Oct 31 18:45:02.148546 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.148531    4484 scope.go:122] "RemoveContainer" containerID="d38d25e9a859776ca585897af7ee6a02ab4c8f2d18481d7a96a17020ae2761ed"
Oct 31 18:45:02.828615 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:02.828586    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:03.033149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.033102    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:03.157851 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.157818    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb" event={"ID":"ce9c46c9-744f-48fd-85ac-43415d1f561c","Type":"ContainerStarted","Data":"198be0793af6f918719e9d773df77ed749d4ad69796170eeb03b071c81d5c78f"}
Oct 31 18:45:03.161029 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.161002    4484 generic.go:356] "Generic (PLEG): container finished" podID="83bf7c50-4e6c-4787-8b53-79b480ad095f" containerID="3c1c34d69756f78c7ab9cb7b0ac31e00f70c0a1837ab43108037ad100cd59af8" exitCode=60
Oct 31 18:45:03.161170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.161092    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l" event={"ID":"83bf7c50-4e6c-4787-8b53-79b480ad095f","Type":"ContainerDied","Data":"3c1c34d69756f78c7ab9cb7b0ac31e00f70c0a1837ab43108037ad100cd59af8"}
Oct 31 18:45:03.161170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.161158    4484 scope.go:122] "RemoveContainer" containerID="b6dfe7f36e1e64fc7e31ffc760ba2b61d82535539be8f2b9f84fee22bda6840b"
Oct 31 18:45:03.161636 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.161460    4484 scope.go:122] "RemoveContainer" containerID="3c1c34d69756f78c7ab9cb7b0ac31e00f70c0a1837ab43108037ad100cd59af8"
Oct 31 18:45:03.161837 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:03.161814    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l_odh-ai-gateway-infra(83bf7c50-4e6c-4787-8b53-79b480ad095f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f"
Oct 31 18:45:03.164726 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.164698    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d" event={"ID":"21cc53ff-5b7d-4c58-8bd2-aacecde51029","Type":"ContainerStarted","Data":"e96db6a7093a47416cb7578f5bd782e94ea597cdfaecf696ddb8fe991f9a8bcb"}
Oct 31 18:45:03.170773 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.169040    4484 generic.go:356] "Generic (PLEG): container finished" podID="d26d5481-5858-4a2d-8bdc-067e04d73052" containerID="20edb18f028ad730507bff8a3f90180f9b16a0082e4948ebfa4f9b81a8e94328" exitCode=60
Oct 31 18:45:03.170773 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.169165    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp" event={"ID":"d26d5481-5858-4a2d-8bdc-067e04d73052","Type":"ContainerDied","Data":"20edb18f028ad730507bff8a3f90180f9b16a0082e4948ebfa4f9b81a8e94328"}
Oct 31 18:45:03.170773 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.169422    4484 scope.go:122] "RemoveContainer" containerID="20edb18f028ad730507bff8a3f90180f9b16a0082e4948ebfa4f9b81a8e94328"
Oct 31 18:45:03.170773 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:03.169802    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp_odh-ai-gateway-infra(d26d5481-5858-4a2d-8bdc-067e04d73052)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052"
Oct 31 18:45:03.216385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.216129    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"]
Oct 31 18:45:03.237785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.237754    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"
Oct 31 18:45:03.262217 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.262192    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"]
Oct 31 18:45:03.290998 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.290968    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cd7d17\""
Oct 31 18:45:03.292626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.292589    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cd7d17-dockercfg-xp6q5\""
Oct 31 18:45:03.296110 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.296064    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tph87\" (UniqueName: \"kubernetes.io/projected/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-kube-api-access-tph87\") pod \"payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v\" (UID: \"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"
Oct 31 18:45:03.296358 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.296332    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v\" (UID: \"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"
Oct 31 18:45:03.303513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.300634    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:03.360581 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.359878    4484 scope.go:122] "RemoveContainer" containerID="d38d25e9a859776ca585897af7ee6a02ab4c8f2d18481d7a96a17020ae2761ed"
Oct 31 18:45:03.378510 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.377409    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb" podStartSLOduration=3.377394232 podStartE2EDuration="3.377394232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:03.29257007 +0000 UTC m=+1449.439019934" watchObservedRunningTime="2026-10-31 18:45:03.377394232 +0000 UTC m=+1449.523844137"
Oct 31 18:45:03.380064 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.380038    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"]
Oct 31 18:45:03.400030 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.399723    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"
Oct 31 18:45:03.401136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.399723    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tph87\" (UniqueName: \"kubernetes.io/projected/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-kube-api-access-tph87\") pod \"payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v\" (UID: \"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"
Oct 31 18:45:03.401136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.400281    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v\" (UID: \"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"
Oct 31 18:45:03.401642 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.401380    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v\" (UID: \"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"
Oct 31 18:45:03.413116 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.413064    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d" podStartSLOduration=3.413047438 podStartE2EDuration="3.413047438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:03.409001243 +0000 UTC m=+1449.555451107" watchObservedRunningTime="2026-10-31 18:45:03.413047438 +0000 UTC m=+1449.559497301"
Oct 31 18:45:03.431644 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.431611    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"]
Oct 31 18:45:03.456967 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.456926    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tph87\" (UniqueName: \"kubernetes.io/projected/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-kube-api-access-tph87\") pod \"payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v\" (UID: \"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"
Oct 31 18:45:03.502304 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.502266    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5mwr\" (UniqueName: \"kubernetes.io/projected/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-kube-api-access-l5mwr\") pod \"payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l\" (UID: \"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6\") " pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"
Oct 31 18:45:03.502569 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.502335    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l\" (UID: \"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6\") " pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"
Oct 31 18:45:03.604230 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.604140    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5mwr\" (UniqueName: \"kubernetes.io/projected/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-kube-api-access-l5mwr\") pod \"payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l\" (UID: \"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6\") " pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"
Oct 31 18:45:03.604230 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.604183    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l\" (UID: \"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6\") " pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"
Oct 31 18:45:03.604994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.604974    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l\" (UID: \"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6\") " pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"
Oct 31 18:45:03.661911 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.661877    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5mwr\" (UniqueName: \"kubernetes.io/projected/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-kube-api-access-l5mwr\") pod \"payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l\" (UID: \"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6\") " pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"
Oct 31 18:45:03.663701 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.663677    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"
Oct 31 18:45:03.723280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:03.723214    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"
Oct 31 18:45:04.043196 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:45:04.043170    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d2af34_9c5e_4e1b_8d1d_404259ea9ef9.slice/crio-e0985829ba4ae763be4035eb5ff5f0422a7734465b66da342dac0d9f53901427 WatchSource:0}: Error finding container e0985829ba4ae763be4035eb5ff5f0422a7734465b66da342dac0d9f53901427: Status 404 returned error can't find the container with id e0985829ba4ae763be4035eb5ff5f0422a7734465b66da342dac0d9f53901427
Oct 31 18:45:04.043528 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.043508    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"]
Oct 31 18:45:04.126594 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.126565    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"]
Oct 31 18:45:04.129855 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:45:04.129830    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71ab3c6_adf2_41c5_a4b1_7e7aad8256c6.slice/crio-e0238eaeda94220c905aa965664cbac80317bd25ab047e433af79746729519e6 WatchSource:0}: Error finding container e0238eaeda94220c905aa965664cbac80317bd25ab047e433af79746729519e6: Status 404 returned error can't find the container with id e0238eaeda94220c905aa965664cbac80317bd25ab047e433af79746729519e6
Oct 31 18:45:04.179567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.179522    4484 generic.go:356] "Generic (PLEG): container finished" podID="ce9c46c9-744f-48fd-85ac-43415d1f561c" containerID="198be0793af6f918719e9d773df77ed749d4ad69796170eeb03b071c81d5c78f" exitCode=60
Oct 31 18:45:04.180057 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.180030    4484 scope.go:122] "RemoveContainer" containerID="198be0793af6f918719e9d773df77ed749d4ad69796170eeb03b071c81d5c78f"
Oct 31 18:45:04.180293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.180270    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb" event={"ID":"ce9c46c9-744f-48fd-85ac-43415d1f561c","Type":"ContainerDied","Data":"198be0793af6f918719e9d773df77ed749d4ad69796170eeb03b071c81d5c78f"}
Oct 31 18:45:04.180361 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.180338    4484 scope.go:122] "RemoveContainer" containerID="42158d348125118d2aaec562847788a97fb4763b13927e5e3ebdaf19b247fb84"
Oct 31 18:45:04.180771 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:04.180687    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb_odh-ai-gateway-infra(ce9c46c9-744f-48fd-85ac-43415d1f561c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c"
Oct 31 18:45:04.182806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.182789    4484 scope.go:122] "RemoveContainer" containerID="3c1c34d69756f78c7ab9cb7b0ac31e00f70c0a1837ab43108037ad100cd59af8"
Oct 31 18:45:04.183084 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:04.183068    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l_odh-ai-gateway-infra(83bf7c50-4e6c-4787-8b53-79b480ad095f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f"
Oct 31 18:45:04.184758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.184729    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v" event={"ID":"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9","Type":"ContainerStarted","Data":"612703de762cdc23f0157954e101ec5346697f9fd7f7e993d6c741afe43bf31a"}
Oct 31 18:45:04.184865 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.184766    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v" event={"ID":"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9","Type":"ContainerStarted","Data":"e0985829ba4ae763be4035eb5ff5f0422a7734465b66da342dac0d9f53901427"}
Oct 31 18:45:04.184865 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.184820    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"
Oct 31 18:45:04.186129 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.186105    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l" event={"ID":"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6","Type":"ContainerStarted","Data":"e0238eaeda94220c905aa965664cbac80317bd25ab047e433af79746729519e6"}
Oct 31 18:45:04.188893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.188867    4484 generic.go:356] "Generic (PLEG): container finished" podID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" containerID="e96db6a7093a47416cb7578f5bd782e94ea597cdfaecf696ddb8fe991f9a8bcb" exitCode=60
Oct 31 18:45:04.188984 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.188945    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d" event={"ID":"21cc53ff-5b7d-4c58-8bd2-aacecde51029","Type":"ContainerDied","Data":"e96db6a7093a47416cb7578f5bd782e94ea597cdfaecf696ddb8fe991f9a8bcb"}
Oct 31 18:45:04.189162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.189144    4484 scope.go:122] "RemoveContainer" containerID="e96db6a7093a47416cb7578f5bd782e94ea597cdfaecf696ddb8fe991f9a8bcb"
Oct 31 18:45:04.189427 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:04.189407    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891205-q8g4d_odh-ai-gateway-infra(21cc53ff-5b7d-4c58-8bd2-aacecde51029)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029"
Oct 31 18:45:04.191151 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.191135    4484 scope.go:122] "RemoveContainer" containerID="20edb18f028ad730507bff8a3f90180f9b16a0082e4948ebfa4f9b81a8e94328"
Oct 31 18:45:04.191389 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:04.191372    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp_odh-ai-gateway-infra(d26d5481-5858-4a2d-8bdc-067e04d73052)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052"
Oct 31 18:45:04.196677 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.196657    4484 scope.go:122] "RemoveContainer" containerID="3ee6510509621dbb0a9a5dae32679e1bdd3f87ccdb4c2ecc7126877958eb61da"
Oct 31 18:45:04.528018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:04.527968    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v" podStartSLOduration=1.527954787 podStartE2EDuration="1.527954787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:04.525550629 +0000 UTC m=+1450.672000492" watchObservedRunningTime="2026-10-31 18:45:04.527954787 +0000 UTC m=+1450.674404665"
Oct 31 18:45:05.200562 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:05.200534    4484 scope.go:122] "RemoveContainer" containerID="198be0793af6f918719e9d773df77ed749d4ad69796170eeb03b071c81d5c78f"
Oct 31 18:45:05.201017 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:05.200905    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb_odh-ai-gateway-infra(ce9c46c9-744f-48fd-85ac-43415d1f561c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c"
Oct 31 18:45:05.201892 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:05.201867    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l" event={"ID":"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6","Type":"ContainerStarted","Data":"e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e"}
Oct 31 18:45:05.202033 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:05.201983    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"
Oct 31 18:45:05.203776 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:05.203755    4484 scope.go:122] "RemoveContainer" containerID="e96db6a7093a47416cb7578f5bd782e94ea597cdfaecf696ddb8fe991f9a8bcb"
Oct 31 18:45:05.204010 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:05.203985    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891205-q8g4d_odh-ai-gateway-infra(21cc53ff-5b7d-4c58-8bd2-aacecde51029)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029"
Oct 31 18:45:05.546599 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:05.546550    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l" podStartSLOduration=2.546537266 podStartE2EDuration="2.546537266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:05.544597367 +0000 UTC m=+1451.691047231" watchObservedRunningTime="2026-10-31 18:45:05.546537266 +0000 UTC m=+1451.692987148"
Oct 31 18:45:08.158011 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:08.157954    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:10.069606 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:10.069575    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7557-acacde"]
Oct 31 18:45:10.075642 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:10.075620    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7557-acacde"
Oct 31 18:45:10.129750 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:10.129726    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bg2\""
Oct 31 18:45:10.131025 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:10.131001    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7557-acacde"]
Oct 31 18:45:10.153629 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:10.153598    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:45:10.197010 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:10.196982    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbhvh\" (UniqueName: \"kubernetes.io/projected/2d847b5c-e9ca-45dc-ba51-de3e40682c4f-kube-api-access-tbhvh\") pod \"test-curl-7557-acacde\" (UID: \"2d847b5c-e9ca-45dc-ba51-de3e40682c4f\") " pod="opendatahub/test-curl-7557-acacde"
Oct 31 18:45:10.298648 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:10.298607    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbhvh\" (UniqueName: \"kubernetes.io/projected/2d847b5c-e9ca-45dc-ba51-de3e40682c4f-kube-api-access-tbhvh\") pod \"test-curl-7557-acacde\" (UID: \"2d847b5c-e9ca-45dc-ba51-de3e40682c4f\") " pod="opendatahub/test-curl-7557-acacde"
Oct 31 18:45:10.355877 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:10.355802    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbhvh\" (UniqueName: \"kubernetes.io/projected/2d847b5c-e9ca-45dc-ba51-de3e40682c4f-kube-api-access-tbhvh\") pod \"test-curl-7557-acacde\" (UID: \"2d847b5c-e9ca-45dc-ba51-de3e40682c4f\") " pod="opendatahub/test-curl-7557-acacde"
Oct 31 18:45:10.389340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:10.389307    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7557-acacde"
Oct 31 18:45:10.765201 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:45:10.765168    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d847b5c_e9ca_45dc_ba51_de3e40682c4f.slice/crio-919a045d43636a788fe9a2a25c37a8d09e68df905efad03a49a58b27ca9c80bc WatchSource:0}: Error finding container 919a045d43636a788fe9a2a25c37a8d09e68df905efad03a49a58b27ca9c80bc: Status 404 returned error can't find the container with id 919a045d43636a788fe9a2a25c37a8d09e68df905efad03a49a58b27ca9c80bc
Oct 31 18:45:10.765316 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:10.765212    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7557-acacde"]
Oct 31 18:45:11.257284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:11.257244    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7557-acacde" event={"ID":"2d847b5c-e9ca-45dc-ba51-de3e40682c4f","Type":"ContainerStarted","Data":"919a045d43636a788fe9a2a25c37a8d09e68df905efad03a49a58b27ca9c80bc"}
Oct 31 18:45:13.050095 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:13.050060    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"]
Oct 31 18:45:13.060886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:13.060860    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872c5b28-rhbmz"]
Oct 31 18:45:13.091328 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:13.091292    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:13.273817 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:13.273783    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:13.768890 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:13.768857    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:13.994064 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:13.994039    4484 scope.go:122] "RemoveContainer" containerID="20edb18f028ad730507bff8a3f90180f9b16a0082e4948ebfa4f9b81a8e94328"
Oct 31 18:45:13.994399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:13.994379    4484 scope.go:122] "RemoveContainer" containerID="3c1c34d69756f78c7ab9cb7b0ac31e00f70c0a1837ab43108037ad100cd59af8"
Oct 31 18:45:13.999273 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:13.999251    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424b4a98-595d-4edb-ae17-a4915eb278c2" path="/var/lib/kubelet/pods/424b4a98-595d-4edb-ae17-a4915eb278c2/volumes"
Oct 31 18:45:14.065085 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:14.064796    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:14.285039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:14.285011    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7557-acacde" event={"ID":"2d847b5c-e9ca-45dc-ba51-de3e40682c4f","Type":"ContainerStarted","Data":"e7b9d487b8ab385350c83d9e65119b764c25c93100e5244df53e4fbd7b79ac75"}
Oct 31 18:45:14.400277 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:14.400226    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7557-acacde" podStartSLOduration=1.6924097040000001 podStartE2EDuration="4.400212876s" totalImagesPullingTime="2.707803172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:14.397200117 +0000 UTC m=+1460.543649980" watchObservedRunningTime="2026-10-31 18:45:14.400212876 +0000 UTC m=+1460.546662739"
Oct 31 18:45:14.987531 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:14.987505    4484 scope.go:122] "RemoveContainer" containerID="198be0793af6f918719e9d773df77ed749d4ad69796170eeb03b071c81d5c78f"
Oct 31 18:45:15.206865 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:15.206841    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"
Oct 31 18:45:15.298846 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:15.298811    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb" event={"ID":"ce9c46c9-744f-48fd-85ac-43415d1f561c","Type":"ContainerStarted","Data":"bf3d1be9daf3b60589d22caa5f61f7ba2ab77dc01b4c4c1f37da3e1cf0daacd7"}
Oct 31 18:45:15.304331 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:15.304302    4484 generic.go:356] "Generic (PLEG): container finished" podID="83bf7c50-4e6c-4787-8b53-79b480ad095f" containerID="1ac33f34b3c63af967f760bdbe579b01a503bbe6c911508ab0ac2fe2fb4b879b" exitCode=60
Oct 31 18:45:15.304558 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:15.304399    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l" event={"ID":"83bf7c50-4e6c-4787-8b53-79b480ad095f","Type":"ContainerDied","Data":"1ac33f34b3c63af967f760bdbe579b01a503bbe6c911508ab0ac2fe2fb4b879b"}
Oct 31 18:45:15.304906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:15.304884    4484 scope.go:122] "RemoveContainer" containerID="3c1c34d69756f78c7ab9cb7b0ac31e00f70c0a1837ab43108037ad100cd59af8"
Oct 31 18:45:15.305018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:15.304981    4484 scope.go:122] "RemoveContainer" containerID="1ac33f34b3c63af967f760bdbe579b01a503bbe6c911508ab0ac2fe2fb4b879b"
Oct 31 18:45:15.305296 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:15.305263    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l_odh-ai-gateway-infra(83bf7c50-4e6c-4787-8b53-79b480ad095f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f"
Oct 31 18:45:15.307875 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:15.307853    4484 generic.go:356] "Generic (PLEG): container finished" podID="d26d5481-5858-4a2d-8bdc-067e04d73052" containerID="c5be481caaa9a664a0cf0ea5c639ee13a449ade753a5a7de4abc9c8978385d92" exitCode=60
Oct 31 18:45:15.307963 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:15.307921    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp" event={"ID":"d26d5481-5858-4a2d-8bdc-067e04d73052","Type":"ContainerDied","Data":"c5be481caaa9a664a0cf0ea5c639ee13a449ade753a5a7de4abc9c8978385d92"}
Oct 31 18:45:15.308355 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:15.308341    4484 scope.go:122] "RemoveContainer" containerID="c5be481caaa9a664a0cf0ea5c639ee13a449ade753a5a7de4abc9c8978385d92"
Oct 31 18:45:15.308632 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:15.308604    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp_odh-ai-gateway-infra(d26d5481-5858-4a2d-8bdc-067e04d73052)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052"
Oct 31 18:45:15.319382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:15.319364    4484 scope.go:122] "RemoveContainer" containerID="20edb18f028ad730507bff8a3f90180f9b16a0082e4948ebfa4f9b81a8e94328"
Oct 31 18:45:15.871431 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:15.871397    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7557-acacde"]
Oct 31 18:45:15.875256 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:15.875229    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7557-acacde"]
Oct 31 18:45:16.222607 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.222531    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"
Oct 31 18:45:16.318824 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.318792    4484 generic.go:356] "Generic (PLEG): container finished" podID="ce9c46c9-744f-48fd-85ac-43415d1f561c" containerID="bf3d1be9daf3b60589d22caa5f61f7ba2ab77dc01b4c4c1f37da3e1cf0daacd7" exitCode=60
Oct 31 18:45:16.320579 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.318873    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb" event={"ID":"ce9c46c9-744f-48fd-85ac-43415d1f561c","Type":"ContainerDied","Data":"bf3d1be9daf3b60589d22caa5f61f7ba2ab77dc01b4c4c1f37da3e1cf0daacd7"}
Oct 31 18:45:16.320579 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.318939    4484 scope.go:122] "RemoveContainer" containerID="198be0793af6f918719e9d773df77ed749d4ad69796170eeb03b071c81d5c78f"
Oct 31 18:45:16.320579 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.319279    4484 scope.go:122] "RemoveContainer" containerID="bf3d1be9daf3b60589d22caa5f61f7ba2ab77dc01b4c4c1f37da3e1cf0daacd7"
Oct 31 18:45:16.320579 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:16.319652    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb_odh-ai-gateway-infra(ce9c46c9-744f-48fd-85ac-43415d1f561c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c"
Oct 31 18:45:16.324546 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.324525    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7557-acacde" podUID="2d847b5c-e9ca-45dc-ba51-de3e40682c4f" containerName="test-curl-7557-acacde" containerID="cri-o://e7b9d487b8ab385350c83d9e65119b764c25c93100e5244df53e4fbd7b79ac75" gracePeriod=2
Oct 31 18:45:16.448572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.448538    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"]
Oct 31 18:45:16.565235 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.565199    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"]
Oct 31 18:45:16.610847 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.610826    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"
Oct 31 18:45:16.678885 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.678854    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"]
Oct 31 18:45:16.701539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.701508    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26d5481-5858-4a2d-8bdc-067e04d73052-kube-api-access-kfrvh\" (UniqueName: \"kubernetes.io/projected/d26d5481-5858-4a2d-8bdc-067e04d73052-kube-api-access-kfrvh\") pod \"d26d5481-5858-4a2d-8bdc-067e04d73052\" (UID: \"d26d5481-5858-4a2d-8bdc-067e04d73052\") "
Oct 31 18:45:16.701841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.701804    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26d5481-5858-4a2d-8bdc-067e04d73052-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26d5481-5858-4a2d-8bdc-067e04d73052-maas-api-service-ca\") pod \"d26d5481-5858-4a2d-8bdc-067e04d73052\" (UID: \"d26d5481-5858-4a2d-8bdc-067e04d73052\") "
Oct 31 18:45:16.702315 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.702290    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26d5481-5858-4a2d-8bdc-067e04d73052-maas-api-service-ca" pod "d26d5481-5858-4a2d-8bdc-067e04d73052" (UID: "d26d5481-5858-4a2d-8bdc-067e04d73052"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:45:16.703671 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.703652    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26d5481-5858-4a2d-8bdc-067e04d73052-kube-api-access-kfrvh" pod "d26d5481-5858-4a2d-8bdc-067e04d73052" (UID: "d26d5481-5858-4a2d-8bdc-067e04d73052"). InnerVolumeSpecName "kube-api-access-kfrvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:16.760697 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.760669    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"]
Oct 31 18:45:16.761377 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.761356    4484 cpu_manager.go:414] "containerMap: removing container" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052" containerName="cleanup"
Oct 31 18:45:16.761377 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.761374    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052" containerName="cleanup"
Oct 31 18:45:16.761630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.761397    4484 cpu_manager.go:414] "containerMap: removing container" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052" containerName="cleanup"
Oct 31 18:45:16.761630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.761404    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052" containerName="cleanup"
Oct 31 18:45:16.761630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.761436    4484 cpu_manager.go:414] "containerMap: removing container" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052" containerName="cleanup"
Oct 31 18:45:16.761630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.761444    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052" containerName="cleanup"
Oct 31 18:45:16.761630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.761467    4484 cpu_manager.go:414] "containerMap: removing container" podUID="2d847b5c-e9ca-45dc-ba51-de3e40682c4f" containerName="test-curl-7557-acacde"
Oct 31 18:45:16.761630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.761490    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d847b5c-e9ca-45dc-ba51-de3e40682c4f" containerName="test-curl-7557-acacde"
Oct 31 18:45:16.761902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.761658    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052" containerName="cleanup"
Oct 31 18:45:16.761902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.761674    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d847b5c-e9ca-45dc-ba51-de3e40682c4f" containerName="test-curl-7557-acacde"
Oct 31 18:45:16.761902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.761695    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052" containerName="cleanup"
Oct 31 18:45:16.765717 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.765658    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:16.805153 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.805116    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzj64\" (UniqueName: \"kubernetes.io/projected/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-kube-api-access-jzj64\") pod \"maas-api-revoke-keys-e2e-mt-3417b822-x86lr\" (UID: \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:16.805293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.805237    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3417b822-x86lr\" (UID: \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:16.805293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.805286    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3417b822-x86lr\" (UID: \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:16.805620 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.805600    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26d5481-5858-4a2d-8bdc-067e04d73052-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:16.805692 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.805626    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfrvh\" (UniqueName: \"kubernetes.io/projected/d26d5481-5858-4a2d-8bdc-067e04d73052-kube-api-access-kfrvh\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:16.819421 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.819231    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 18:45:16.820677 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.820653    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"]
Oct 31 18:45:16.884159 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.884138    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"
Oct 31 18:45:16.906179 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.906155    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bf7c50-4e6c-4787-8b53-79b480ad095f-kube-api-access-xdrqt\" (UniqueName: \"kubernetes.io/projected/83bf7c50-4e6c-4787-8b53-79b480ad095f-kube-api-access-xdrqt\") pod \"83bf7c50-4e6c-4787-8b53-79b480ad095f\" (UID: \"83bf7c50-4e6c-4787-8b53-79b480ad095f\") "
Oct 31 18:45:16.906305 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.906198    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83bf7c50-4e6c-4787-8b53-79b480ad095f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bf7c50-4e6c-4787-8b53-79b480ad095f-maas-api-service-ca\") pod \"83bf7c50-4e6c-4787-8b53-79b480ad095f\" (UID: \"83bf7c50-4e6c-4787-8b53-79b480ad095f\") "
Oct 31 18:45:16.906527 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.906506    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzj64\" (UniqueName: \"kubernetes.io/projected/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-kube-api-access-jzj64\") pod \"maas-api-revoke-keys-e2e-mt-3417b822-x86lr\" (UID: \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:16.906594 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.906577    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3417b822-x86lr\" (UID: \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:16.906657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.906626    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3417b822-x86lr\" (UID: \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:16.906657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.906634    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83bf7c50-4e6c-4787-8b53-79b480ad095f-maas-api-service-ca" pod "83bf7c50-4e6c-4787-8b53-79b480ad095f" (UID: "83bf7c50-4e6c-4787-8b53-79b480ad095f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:45:16.906766 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.906752    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bf7c50-4e6c-4787-8b53-79b480ad095f-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:16.907227 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.907201    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3417b822-x86lr\" (UID: \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:16.908411 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.908393    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bf7c50-4e6c-4787-8b53-79b480ad095f-kube-api-access-xdrqt" pod "83bf7c50-4e6c-4787-8b53-79b480ad095f" (UID: "83bf7c50-4e6c-4787-8b53-79b480ad095f"). InnerVolumeSpecName "kube-api-access-xdrqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:16.908895 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.908879    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3417b822-x86lr\" (UID: \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:16.963726 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:16.963697    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzj64\" (UniqueName: \"kubernetes.io/projected/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-kube-api-access-jzj64\") pod \"maas-api-revoke-keys-e2e-mt-3417b822-x86lr\" (UID: \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:17.008127 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.008096    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdrqt\" (UniqueName: \"kubernetes.io/projected/83bf7c50-4e6c-4787-8b53-79b480ad095f-kube-api-access-xdrqt\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:17.101105 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.101016    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:17.333597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.333567    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"
Oct 31 18:45:17.334081 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.333595    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp" event={"ID":"d26d5481-5858-4a2d-8bdc-067e04d73052","Type":"ContainerDied","Data":"ef56e019b56409160ea8f532e03a2419cad525fb529edf0007d78a2d8b437f6f"}
Oct 31 18:45:17.334081 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.333698    4484 scope.go:122] "RemoveContainer" containerID="c5be481caaa9a664a0cf0ea5c639ee13a449ade753a5a7de4abc9c8978385d92"
Oct 31 18:45:17.339723 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.339688    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l" event={"ID":"83bf7c50-4e6c-4787-8b53-79b480ad095f","Type":"ContainerDied","Data":"45fb3bfbb9c174f66c791568bcf51d553462a65317c12d039772b6e4f9ab6ffd"}
Oct 31 18:45:17.339812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.339703    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"
Oct 31 18:45:17.346380 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.346361    4484 scope.go:122] "RemoveContainer" containerID="1ac33f34b3c63af967f760bdbe579b01a503bbe6c911508ab0ac2fe2fb4b879b"
Oct 31 18:45:17.436577 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.436546    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:17.488578 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:45:17.488536    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a1bd12_cbc4_43af_bae3_cb29092bf68f.slice/crio-3fca30f6477e5f061d96fc75886b70f917e15d46dbe43227ad2c02b5e07f4d86 WatchSource:0}: Error finding container 3fca30f6477e5f061d96fc75886b70f917e15d46dbe43227ad2c02b5e07f4d86: Status 404 returned error can't find the container with id 3fca30f6477e5f061d96fc75886b70f917e15d46dbe43227ad2c02b5e07f4d86
Oct 31 18:45:17.489789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.489542    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"]
Oct 31 18:45:17.505075 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.505051    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"]
Oct 31 18:45:17.508727 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.508709    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891205-dgh4l"]
Oct 31 18:45:17.532710 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.532690    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"
Oct 31 18:45:17.616694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.616618    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9c46c9-744f-48fd-85ac-43415d1f561c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9c46c9-744f-48fd-85ac-43415d1f561c-maas-api-service-ca\") pod \"ce9c46c9-744f-48fd-85ac-43415d1f561c\" (UID: \"ce9c46c9-744f-48fd-85ac-43415d1f561c\") "
Oct 31 18:45:17.616817 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.616752    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9c46c9-744f-48fd-85ac-43415d1f561c-kube-api-access-n9sqw\" (UniqueName: \"kubernetes.io/projected/ce9c46c9-744f-48fd-85ac-43415d1f561c-kube-api-access-n9sqw\") pod \"ce9c46c9-744f-48fd-85ac-43415d1f561c\" (UID: \"ce9c46c9-744f-48fd-85ac-43415d1f561c\") "
Oct 31 18:45:17.617035 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.617012    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9c46c9-744f-48fd-85ac-43415d1f561c-maas-api-service-ca" pod "ce9c46c9-744f-48fd-85ac-43415d1f561c" (UID: "ce9c46c9-744f-48fd-85ac-43415d1f561c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:45:17.617440 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.617423    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9c46c9-744f-48fd-85ac-43415d1f561c-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:17.618985 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.618965    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9c46c9-744f-48fd-85ac-43415d1f561c-kube-api-access-n9sqw" pod "ce9c46c9-744f-48fd-85ac-43415d1f561c" (UID: "ce9c46c9-744f-48fd-85ac-43415d1f561c"). InnerVolumeSpecName "kube-api-access-n9sqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:17.626117 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.626091    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"]
Oct 31 18:45:17.645559 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.645535    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-373109-29891205-v42pp"]
Oct 31 18:45:17.718280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.718254    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9sqw\" (UniqueName: \"kubernetes.io/projected/ce9c46c9-744f-48fd-85ac-43415d1f561c-kube-api-access-n9sqw\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:17.987111 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.987087    4484 scope.go:122] "RemoveContainer" containerID="e96db6a7093a47416cb7578f5bd782e94ea597cdfaecf696ddb8fe991f9a8bcb"
Oct 31 18:45:17.996373 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.996344    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f" path="/var/lib/kubelet/pods/83bf7c50-4e6c-4787-8b53-79b480ad095f/volumes"
Oct 31 18:45:17.996961 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:17.996942    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052" path="/var/lib/kubelet/pods/d26d5481-5858-4a2d-8bdc-067e04d73052/volumes"
Oct 31 18:45:18.347637 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.347611    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"
Oct 31 18:45:18.347637 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.347621    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb" event={"ID":"ce9c46c9-744f-48fd-85ac-43415d1f561c","Type":"ContainerDied","Data":"b8e4c7559e4e6536a0674a92d61a47b36efba3d0f180b265adb2cdf0c270dd5b"}
Oct 31 18:45:18.348049 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.347684    4484 scope.go:122] "RemoveContainer" containerID="bf3d1be9daf3b60589d22caa5f61f7ba2ab77dc01b4c4c1f37da3e1cf0daacd7"
Oct 31 18:45:18.349347 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.349318    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:18.351465 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.351441    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr" event={"ID":"c0a1bd12-cbc4-43af-bae3-cb29092bf68f","Type":"ContainerStarted","Data":"040161d0211a2dbe56dbd0008bfe36617331ad9c31ecddc120f7083686fdb94a"}
Oct 31 18:45:18.351588 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.351525    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr" event={"ID":"c0a1bd12-cbc4-43af-bae3-cb29092bf68f","Type":"ContainerStarted","Data":"3fca30f6477e5f061d96fc75886b70f917e15d46dbe43227ad2c02b5e07f4d86"}
Oct 31 18:45:18.353407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.353388    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d" event={"ID":"21cc53ff-5b7d-4c58-8bd2-aacecde51029","Type":"ContainerStarted","Data":"6e8cfa4a26d5a346f4548a044544033c5e432fbc1a5a2424bea17d53bfb0e2b8"}
Oct 31 18:45:18.523063 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.522952    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:18.566528 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.566502    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7557-acacde"
Oct 31 18:45:18.629733 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.629646    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d847b5c-e9ca-45dc-ba51-de3e40682c4f-kube-api-access-tbhvh\" (UniqueName: \"kubernetes.io/projected/2d847b5c-e9ca-45dc-ba51-de3e40682c4f-kube-api-access-tbhvh\") pod \"2d847b5c-e9ca-45dc-ba51-de3e40682c4f\" (UID: \"2d847b5c-e9ca-45dc-ba51-de3e40682c4f\") "
Oct 31 18:45:18.632290 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.632263    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d847b5c-e9ca-45dc-ba51-de3e40682c4f-kube-api-access-tbhvh" pod "2d847b5c-e9ca-45dc-ba51-de3e40682c4f" (UID: "2d847b5c-e9ca-45dc-ba51-de3e40682c4f"). InnerVolumeSpecName "kube-api-access-tbhvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:18.635605 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.635563    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr" podStartSLOduration=2.63555068 podStartE2EDuration="2.63555068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:18.634099317 +0000 UTC m=+1464.780549180" watchObservedRunningTime="2026-10-31 18:45:18.63555068 +0000 UTC m=+1464.782000597"
Oct 31 18:45:18.696609 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.696567    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:18.730998 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.730971    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbhvh\" (UniqueName: \"kubernetes.io/projected/2d847b5c-e9ca-45dc-ba51-de3e40682c4f-kube-api-access-tbhvh\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:18.747517 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.747486    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"]
Oct 31 18:45:18.756507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:18.756456    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891205-4xfkb"]
Oct 31 18:45:19.362355 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.362314    4484 generic.go:356] "Generic (PLEG): container finished" podID="c0a1bd12-cbc4-43af-bae3-cb29092bf68f" containerID="040161d0211a2dbe56dbd0008bfe36617331ad9c31ecddc120f7083686fdb94a" exitCode=0
Oct 31 18:45:19.362815 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.362399    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr" event={"ID":"c0a1bd12-cbc4-43af-bae3-cb29092bf68f","Type":"ContainerDied","Data":"040161d0211a2dbe56dbd0008bfe36617331ad9c31ecddc120f7083686fdb94a"}
Oct 31 18:45:19.364204 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.364183    4484 generic.go:356] "Generic (PLEG): container finished" podID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" containerID="6e8cfa4a26d5a346f4548a044544033c5e432fbc1a5a2424bea17d53bfb0e2b8" exitCode=60
Oct 31 18:45:19.364279 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.364256    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d" event={"ID":"21cc53ff-5b7d-4c58-8bd2-aacecde51029","Type":"ContainerDied","Data":"6e8cfa4a26d5a346f4548a044544033c5e432fbc1a5a2424bea17d53bfb0e2b8"}
Oct 31 18:45:19.364340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.364316    4484 scope.go:122] "RemoveContainer" containerID="e96db6a7093a47416cb7578f5bd782e94ea597cdfaecf696ddb8fe991f9a8bcb"
Oct 31 18:45:19.364687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.364666    4484 scope.go:122] "RemoveContainer" containerID="6e8cfa4a26d5a346f4548a044544033c5e432fbc1a5a2424bea17d53bfb0e2b8"
Oct 31 18:45:19.364983 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:19.364960    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891205-q8g4d_odh-ai-gateway-infra(21cc53ff-5b7d-4c58-8bd2-aacecde51029)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029"
Oct 31 18:45:19.366392 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.366373    4484 generic.go:356] "Generic (PLEG): container finished" podID="2d847b5c-e9ca-45dc-ba51-de3e40682c4f" containerID="e7b9d487b8ab385350c83d9e65119b764c25c93100e5244df53e4fbd7b79ac75" exitCode=137
Oct 31 18:45:19.366487 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.366406    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7557-acacde"
Oct 31 18:45:19.378000 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.377982    4484 scope.go:122] "RemoveContainer" containerID="e7b9d487b8ab385350c83d9e65119b764c25c93100e5244df53e4fbd7b79ac75"
Oct 31 18:45:19.386994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.386974    4484 scope.go:122] "RemoveContainer" containerID="e7b9d487b8ab385350c83d9e65119b764c25c93100e5244df53e4fbd7b79ac75"
Oct 31 18:45:19.387249 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:19.387232    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b9d487b8ab385350c83d9e65119b764c25c93100e5244df53e4fbd7b79ac75\": container with ID starting with e7b9d487b8ab385350c83d9e65119b764c25c93100e5244df53e4fbd7b79ac75 not found: ID does not exist" containerID="e7b9d487b8ab385350c83d9e65119b764c25c93100e5244df53e4fbd7b79ac75"
Oct 31 18:45:19.387313 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.387256    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b9d487b8ab385350c83d9e65119b764c25c93100e5244df53e4fbd7b79ac75"} err="failed to get container status \"e7b9d487b8ab385350c83d9e65119b764c25c93100e5244df53e4fbd7b79ac75\": rpc error: code = NotFound desc = could not find container \"e7b9d487b8ab385350c83d9e65119b764c25c93100e5244df53e4fbd7b79ac75\": container with ID starting with e7b9d487b8ab385350c83d9e65119b764c25c93100e5244df53e4fbd7b79ac75 not found: ID does not exist"
Oct 31 18:45:19.533871 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.533841    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"]
Oct 31 18:45:19.651574 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:19.651461    4484 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7557-acacde\" is forbidden: User \"system:node:ip-10-0-142-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-120.us-west-1.compute.internal' and this object" podUID="2d847b5c-e9ca-45dc-ba51-de3e40682c4f" pod="opendatahub/test-curl-7557-acacde"
Oct 31 18:45:19.994668 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.994640    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d847b5c-e9ca-45dc-ba51-de3e40682c4f" path="/var/lib/kubelet/pods/2d847b5c-e9ca-45dc-ba51-de3e40682c4f/volumes"
Oct 31 18:45:19.995057 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:19.995044    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c" path="/var/lib/kubelet/pods/ce9c46c9-744f-48fd-85ac-43415d1f561c/volumes"
Oct 31 18:45:20.083821 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.083781    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:20.582166 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.582140    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:20.584075 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.584059    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"
Oct 31 18:45:20.656874 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.656796    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21cc53ff-5b7d-4c58-8bd2-aacecde51029-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21cc53ff-5b7d-4c58-8bd2-aacecde51029-maas-api-service-ca\") pod \"21cc53ff-5b7d-4c58-8bd2-aacecde51029\" (UID: \"21cc53ff-5b7d-4c58-8bd2-aacecde51029\") "
Oct 31 18:45:20.656874 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.656859    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-openshift-ca\") pod \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\" (UID: \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\") "
Oct 31 18:45:20.657039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.656940    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-xks-ca\") pod \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\" (UID: \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\") "
Oct 31 18:45:20.657039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.656970    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21cc53ff-5b7d-4c58-8bd2-aacecde51029-kube-api-access-r96jc\" (UniqueName: \"kubernetes.io/projected/21cc53ff-5b7d-4c58-8bd2-aacecde51029-kube-api-access-r96jc\") pod \"21cc53ff-5b7d-4c58-8bd2-aacecde51029\" (UID: \"21cc53ff-5b7d-4c58-8bd2-aacecde51029\") "
Oct 31 18:45:20.657150 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.657072    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-kube-api-access-jzj64\" (UniqueName: \"kubernetes.io/projected/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-kube-api-access-jzj64\") pod \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\" (UID: \"c0a1bd12-cbc4-43af-bae3-cb29092bf68f\") "
Oct 31 18:45:20.657267 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.657236    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21cc53ff-5b7d-4c58-8bd2-aacecde51029-maas-api-service-ca" pod "21cc53ff-5b7d-4c58-8bd2-aacecde51029" (UID: "21cc53ff-5b7d-4c58-8bd2-aacecde51029"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:45:20.657357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.657320    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-openshift-ca" pod "c0a1bd12-cbc4-43af-bae3-cb29092bf68f" (UID: "c0a1bd12-cbc4-43af-bae3-cb29092bf68f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:45:20.657816 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.657715    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21cc53ff-5b7d-4c58-8bd2-aacecde51029-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:20.657816 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.657781    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:20.659302 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.659280    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-xks-ca" pod "c0a1bd12-cbc4-43af-bae3-cb29092bf68f" (UID: "c0a1bd12-cbc4-43af-bae3-cb29092bf68f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:45:20.659377 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.659319    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21cc53ff-5b7d-4c58-8bd2-aacecde51029-kube-api-access-r96jc" pod "21cc53ff-5b7d-4c58-8bd2-aacecde51029" (UID: "21cc53ff-5b7d-4c58-8bd2-aacecde51029"). InnerVolumeSpecName "kube-api-access-r96jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:20.659421 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.659369    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-kube-api-access-jzj64" pod "c0a1bd12-cbc4-43af-bae3-cb29092bf68f" (UID: "c0a1bd12-cbc4-43af-bae3-cb29092bf68f"). InnerVolumeSpecName "kube-api-access-jzj64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:20.759544 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.759515    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:20.759544 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.759541    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r96jc\" (UniqueName: \"kubernetes.io/projected/21cc53ff-5b7d-4c58-8bd2-aacecde51029-kube-api-access-r96jc\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:20.759544 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:20.759552    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzj64\" (UniqueName: \"kubernetes.io/projected/c0a1bd12-cbc4-43af-bae3-cb29092bf68f-kube-api-access-jzj64\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:21.387171 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:21.387137    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"
Oct 31 18:45:21.387171 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:21.387162    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr" event={"ID":"c0a1bd12-cbc4-43af-bae3-cb29092bf68f","Type":"ContainerDied","Data":"3fca30f6477e5f061d96fc75886b70f917e15d46dbe43227ad2c02b5e07f4d86"}
Oct 31 18:45:21.387423 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:21.387214    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fca30f6477e5f061d96fc75886b70f917e15d46dbe43227ad2c02b5e07f4d86"
Oct 31 18:45:21.388798 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:21.388775    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"
Oct 31 18:45:21.388941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:21.388813    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d" event={"ID":"21cc53ff-5b7d-4c58-8bd2-aacecde51029","Type":"ContainerDied","Data":"9ef388e99862db5b407562ab38cb953ea918fc2acfbe3eb83209165cb6bed9f2"}
Oct 31 18:45:21.388941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:21.388873    4484 scope.go:122] "RemoveContainer" containerID="6e8cfa4a26d5a346f4548a044544033c5e432fbc1a5a2424bea17d53bfb0e2b8"
Oct 31 18:45:21.555298 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:21.555270    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"]
Oct 31 18:45:21.560601 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:21.560573    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891205-q8g4d"]
Oct 31 18:45:21.595277 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:21.595252    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:21.760990 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:21.760958    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:22.005907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.005865    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" path="/var/lib/kubelet/pods/21cc53ff-5b7d-4c58-8bd2-aacecde51029/volumes"
Oct 31 18:45:22.137543 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.137421    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"]
Oct 31 18:45:22.138294 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138270    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c" containerName="cleanup"
Oct 31 18:45:22.138294 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138294    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138315    4484 cpu_manager.go:414] "containerMap: removing container" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138323    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138336    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138343    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138354    4484 cpu_manager.go:414] "containerMap: removing container" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138360    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138367    4484 cpu_manager.go:414] "containerMap: removing container" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138373    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138387    4484 cpu_manager.go:414] "containerMap: removing container" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138393    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138411    4484 cpu_manager.go:414] "containerMap: removing container" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138418    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138428    4484 cpu_manager.go:414] "containerMap: removing container" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138434    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" containerName="cleanup"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138461    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c0a1bd12-cbc4-43af-bae3-cb29092bf68f" containerName="revoke-keys"
Oct 31 18:45:22.138498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138468    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a1bd12-cbc4-43af-bae3-cb29092bf68f" containerName="revoke-keys"
Oct 31 18:45:22.139197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138658    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26d5481-5858-4a2d-8bdc-067e04d73052" containerName="cleanup"
Oct 31 18:45:22.139197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138673    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c" containerName="cleanup"
Oct 31 18:45:22.139197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138685    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f" containerName="cleanup"
Oct 31 18:45:22.139197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138695    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c" containerName="cleanup"
Oct 31 18:45:22.139197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138703    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" containerName="cleanup"
Oct 31 18:45:22.139197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138709    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" containerName="cleanup"
Oct 31 18:45:22.139197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138717    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f" containerName="cleanup"
Oct 31 18:45:22.139197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138728    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bf7c50-4e6c-4787-8b53-79b480ad095f" containerName="cleanup"
Oct 31 18:45:22.139197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138735    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="21cc53ff-5b7d-4c58-8bd2-aacecde51029" containerName="cleanup"
Oct 31 18:45:22.139197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.138746    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a1bd12-cbc4-43af-bae3-cb29092bf68f" containerName="revoke-keys"
Oct 31 18:45:22.144177 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.144154    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"
Oct 31 18:45:22.176632 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.176605    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt2fd\" (UniqueName: \"kubernetes.io/projected/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-kube-api-access-zt2fd\") pod \"payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4\" (UID: \"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac\") " pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"
Oct 31 18:45:22.176781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.176691    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4\" (UID: \"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac\") " pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"
Oct 31 18:45:22.197431 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.197398    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c21e9e\""
Oct 31 18:45:22.197643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.197517    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c21e9e-dockercfg-m2ksr\""
Oct 31 18:45:22.200484 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.200443    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"]
Oct 31 18:45:22.277998 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.277956    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt2fd\" (UniqueName: \"kubernetes.io/projected/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-kube-api-access-zt2fd\") pod \"payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4\" (UID: \"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac\") " pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"
Oct 31 18:45:22.277998 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.278009    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4\" (UID: \"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac\") " pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"
Oct 31 18:45:22.278724 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.278697    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4\" (UID: \"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac\") " pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"
Oct 31 18:45:22.335407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.335364    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt2fd\" (UniqueName: \"kubernetes.io/projected/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-kube-api-access-zt2fd\") pod \"payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4\" (UID: \"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac\") " pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"
Oct 31 18:45:22.462905 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.462830    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"
Oct 31 18:45:22.836269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:22.836242    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"]
Oct 31 18:45:22.836604 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:45:22.836362    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ef50d6_cfc1_4bb0_8597_ac7d5f8c7eac.slice/crio-3ea05d9fa16d5b99c8c3240153da26202600b20160cffd5d1974e9d7cf18a8f5 WatchSource:0}: Error finding container 3ea05d9fa16d5b99c8c3240153da26202600b20160cffd5d1974e9d7cf18a8f5: Status 404 returned error can't find the container with id 3ea05d9fa16d5b99c8c3240153da26202600b20160cffd5d1974e9d7cf18a8f5
Oct 31 18:45:23.404893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:23.404860    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4" event={"ID":"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac","Type":"ContainerStarted","Data":"bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227"}
Oct 31 18:45:23.405069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:23.404908    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4" event={"ID":"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac","Type":"ContainerStarted","Data":"3ea05d9fa16d5b99c8c3240153da26202600b20160cffd5d1974e9d7cf18a8f5"}
Oct 31 18:45:23.405108 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:23.405081    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"
Oct 31 18:45:23.523645 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:23.523598    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4" podStartSLOduration=1.523585518 podStartE2EDuration="1.523585518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:23.522685599 +0000 UTC m=+1469.669135462" watchObservedRunningTime="2026-10-31 18:45:23.523585518 +0000 UTC m=+1469.670035381"
Oct 31 18:45:23.608467 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:23.608437    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:25.469936 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:25.469902    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:45:25.997332 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:25.997295    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:27.423635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:27.423595    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"]
Oct 31 18:45:27.701630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:27.701535    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:27.899158 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:27.899125    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:28.589367 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:28.589331    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:30.045093 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:30.045060    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"]
Oct 31 18:45:30.095849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:30.095816    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-529f1630-8jtfs"]
Oct 31 18:45:30.851284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:30.851247    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:31.995896 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:31.995861    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d694131f-187f-4bc5-a78c-db7dd7e754de" path="/var/lib/kubelet/pods/d694131f-187f-4bc5-a78c-db7dd7e754de/volumes"
Oct 31 18:45:32.424770 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:32.424689    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l" podUID="97aced5d-a853-4858-bbe7-fbe1088dd7fe" containerName="payload-pre-processing" containerID="cri-o://8c9f188a878e878dafc883dced60d8fd197a7800bd96bb81edd127e3fbb0e42d" gracePeriod=25
Oct 31 18:45:33.242243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:33.242212    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:34.415153 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:34.415124    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"
Oct 31 18:45:35.166338 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:35.166306    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l" podUID="97aced5d-a853-4858-bbe7-fbe1088dd7fe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:45:37.870537 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:37.870436    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:41.054117 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:41.054088    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"]
Oct 31 18:45:41.104443 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:41.104404    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c36c08b-8r5hc"]
Oct 31 18:45:41.997517 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:41.997466    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c422eb47-48de-4d6c-bc9d-812797f4cb26" path="/var/lib/kubelet/pods/c422eb47-48de-4d6c-bc9d-812797f4cb26/volumes"
Oct 31 18:45:42.645043 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:42.644998    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:44.168146 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:44.168109    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:45.166686 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:45.166654    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l" podUID="97aced5d-a853-4858-bbe7-fbe1088dd7fe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 18:45:45.860346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:45.860308    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:47.652730 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.652682    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"]
Oct 31 18:45:47.654229 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.654190    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c" containerName="cleanup"
Oct 31 18:45:47.654229 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.654229    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c" containerName="cleanup"
Oct 31 18:45:47.654604 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.654588    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9c46c9-744f-48fd-85ac-43415d1f561c" containerName="cleanup"
Oct 31 18:45:47.665791 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.665769    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.718598 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.718574    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"]
Oct 31 18:45:47.760528 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.760468    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-maas-api-tls\") pod \"maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.760736 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.760708    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c09c680-68f0-46b1-becd-01129885f5e7-openshift-service-ca\") pod \"maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.760811 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.760781    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-metrics-certs\") pod \"maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.760875 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.760842    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzk9z\" (UniqueName: \"kubernetes.io/projected/3c09c680-68f0-46b1-becd-01129885f5e7-kube-api-access-nzk9z\") pod \"maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.764973 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.764949    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cd7d17-serving-cert\""
Oct 31 18:45:47.823587 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.823553    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:47.862752 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.862712    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzk9z\" (UniqueName: \"kubernetes.io/projected/3c09c680-68f0-46b1-becd-01129885f5e7-kube-api-access-nzk9z\") pod \"maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.862984 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.862868    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-maas-api-tls\") pod \"maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.863060 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.863005    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c09c680-68f0-46b1-becd-01129885f5e7-openshift-service-ca\") pod \"maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.863060 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.863053    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-metrics-certs\") pod \"maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.863741 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.863696    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c09c680-68f0-46b1-becd-01129885f5e7-openshift-service-ca\") pod \"maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.874514 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.871140    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-maas-api-tls\") pod \"maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.874514 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.872428    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-metrics-certs\") pod \"maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.920627 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.920546    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzk9z\" (UniqueName: \"kubernetes.io/projected/3c09c680-68f0-46b1-becd-01129885f5e7-kube-api-access-nzk9z\") pod \"maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.979885 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.979860    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:47.980189 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:47.980166    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:48.135340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:48.135300    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:48.383968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:48.383943    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"]
Oct 31 18:45:48.384093 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:45:48.384019    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c09c680_68f0_46b1_becd_01129885f5e7.slice/crio-5cfcb1070a1362d086df52f447b2d04f432615fd468217fbc79df5bf83e64cde WatchSource:0}: Error finding container 5cfcb1070a1362d086df52f447b2d04f432615fd468217fbc79df5bf83e64cde: Status 404 returned error can't find the container with id 5cfcb1070a1362d086df52f447b2d04f432615fd468217fbc79df5bf83e64cde
Oct 31 18:45:48.482739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:48.482702    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:48.581074 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:48.581042    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds" event={"ID":"3c09c680-68f0-46b1-becd-01129885f5e7","Type":"ContainerStarted","Data":"5cfcb1070a1362d086df52f447b2d04f432615fd468217fbc79df5bf83e64cde"}
Oct 31 18:45:49.589611 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:49.589575    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds" event={"ID":"3c09c680-68f0-46b1-becd-01129885f5e7","Type":"ContainerStarted","Data":"bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9"}
Oct 31 18:45:49.590034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:49.589783    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:49.710377 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:49.710325    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds" podStartSLOduration=1.877655603 podStartE2EDuration="2.710310264s" totalImagesPullingTime="832.654661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:45:49.706787314 +0000 UTC m=+1495.853237177" watchObservedRunningTime="2026-10-31 18:45:49.710310264 +0000 UTC m=+1495.856760127"
Oct 31 18:45:51.665445 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:51.665409    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:52.655944 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:52.655902    4484 generic.go:356] "Generic (PLEG): container finished" podID="97aced5d-a853-4858-bbe7-fbe1088dd7fe" containerID="8c9f188a878e878dafc883dced60d8fd197a7800bd96bb81edd127e3fbb0e42d" exitCode=0
Oct 31 18:45:52.656134 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:52.656048    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l" event={"ID":"97aced5d-a853-4858-bbe7-fbe1088dd7fe","Type":"ContainerDied","Data":"8c9f188a878e878dafc883dced60d8fd197a7800bd96bb81edd127e3fbb0e42d"}
Oct 31 18:45:52.763270 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:52.762626    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"
Oct 31 18:45:52.845034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:52.844998    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97aced5d-a853-4858-bbe7-fbe1088dd7fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97aced5d-a853-4858-bbe7-fbe1088dd7fe-plugins-config-volume\") pod \"97aced5d-a853-4858-bbe7-fbe1088dd7fe\" (UID: \"97aced5d-a853-4858-bbe7-fbe1088dd7fe\") "
Oct 31 18:45:52.845209 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:52.845136    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97aced5d-a853-4858-bbe7-fbe1088dd7fe-kube-api-access-8r2gr\" (UniqueName: \"kubernetes.io/projected/97aced5d-a853-4858-bbe7-fbe1088dd7fe-kube-api-access-8r2gr\") pod \"97aced5d-a853-4858-bbe7-fbe1088dd7fe\" (UID: \"97aced5d-a853-4858-bbe7-fbe1088dd7fe\") "
Oct 31 18:45:52.845503 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:52.845443    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97aced5d-a853-4858-bbe7-fbe1088dd7fe-plugins-config-volume" pod "97aced5d-a853-4858-bbe7-fbe1088dd7fe" (UID: "97aced5d-a853-4858-bbe7-fbe1088dd7fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:45:52.845821 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:52.845801    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97aced5d-a853-4858-bbe7-fbe1088dd7fe-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:52.847547 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:52.847528    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97aced5d-a853-4858-bbe7-fbe1088dd7fe-kube-api-access-8r2gr" pod "97aced5d-a853-4858-bbe7-fbe1088dd7fe" (UID: "97aced5d-a853-4858-bbe7-fbe1088dd7fe"). InnerVolumeSpecName "kube-api-access-8r2gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:45:52.947486 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:52.947448    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r2gr\" (UniqueName: \"kubernetes.io/projected/97aced5d-a853-4858-bbe7-fbe1088dd7fe-kube-api-access-8r2gr\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:45:53.179780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:53.179740    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:53.690245 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:53.690209    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l" event={"ID":"97aced5d-a853-4858-bbe7-fbe1088dd7fe","Type":"ContainerDied","Data":"be44a99fb4d665a338af77e9d6773492586abd6b62ade16774636d0f14dbfdd2"}
Oct 31 18:45:53.690455 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:53.690282    4484 scope.go:122] "RemoveContainer" containerID="8c9f188a878e878dafc883dced60d8fd197a7800bd96bb81edd127e3fbb0e42d"
Oct 31 18:45:53.690455 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:53.690453    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"
Oct 31 18:45:53.862505 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:53.862446    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"]
Oct 31 18:45:53.931833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:53.931706    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-9sc2l"]
Oct 31 18:45:53.995832 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:53.995801    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97aced5d-a853-4858-bbe7-fbe1088dd7fe" path="/var/lib/kubelet/pods/97aced5d-a853-4858-bbe7-fbe1088dd7fe/volumes"
Oct 31 18:45:54.322750 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:54.322677    4484 scope.go:122] "RemoveContainer" containerID="ffcc25b8b511d6f6e9b1c5fda9ff349568f3ac085a8b558d98e490a00bed9489"
Oct 31 18:45:54.332358 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:54.332338    4484 scope.go:122] "RemoveContainer" containerID="a4620d63ef542821ef96ab22c75cb538498cb71220105b82426a4e41ecd18fb2"
Oct 31 18:45:54.422017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:54.421992    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:45:54.435610 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:54.435588    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:45:55.468349 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:45:55.468312    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:45:55.633086 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:55.633060    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:45:57.045769 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:57.045739    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"]
Oct 31 18:45:57.088136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:57.088103    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:45:57.095765 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:57.095739    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84ee3021-cqcnn"]
Oct 31 18:45:57.994795 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:57.994760    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd85448-6da1-473e-857e-d02837627368" path="/var/lib/kubelet/pods/ccd85448-6da1-473e-857e-d02837627368/volumes"
Oct 31 18:45:58.102830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:45:58.102799    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:02.952004 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:02.951966    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:05.552290 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:05.552261    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:46:07.050694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:07.050657    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"]
Oct 31 18:46:07.100570 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:07.100533    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a2fda1-gwk7j"]
Oct 31 18:46:07.646095 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:07.646060    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:07.802974 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:07.802942    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:07.961028 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:07.960928    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:07.995117 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:07.995076    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40380b55-67f1-4f80-9152-f449137dce80" path="/var/lib/kubelet/pods/40380b55-67f1-4f80-9152-f449137dce80/volumes"
Oct 31 18:46:12.778855 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:12.778809    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:16.753581 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:16.753547    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"]
Oct 31 18:46:16.754049 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:16.753828    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp" podUID="5f80435b-76fb-41c8-a6a6-c5d99fe23021" containerName="maas-api" containerID="cri-o://5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76" gracePeriod=30
Oct 31 18:46:16.878132 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:16.878096    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:17.041849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.041823    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:46:17.170039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.169996    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f80435b-76fb-41c8-a6a6-c5d99fe23021-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f80435b-76fb-41c8-a6a6-c5d99fe23021-openshift-service-ca\") pod \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") "
Oct 31 18:46:17.170406 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.170371    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f80435b-76fb-41c8-a6a6-c5d99fe23021-openshift-service-ca" pod "5f80435b-76fb-41c8-a6a6-c5d99fe23021" (UID: "5f80435b-76fb-41c8-a6a6-c5d99fe23021"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:46:17.170582 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.170433    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-maas-api-tls\") pod \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") "
Oct 31 18:46:17.170725 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.170704    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-metrics-certs\") pod \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") "
Oct 31 18:46:17.170792 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.170750    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f80435b-76fb-41c8-a6a6-c5d99fe23021-kube-api-access-6cm6p\" (UniqueName: \"kubernetes.io/projected/5f80435b-76fb-41c8-a6a6-c5d99fe23021-kube-api-access-6cm6p\") pod \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\" (UID: \"5f80435b-76fb-41c8-a6a6-c5d99fe23021\") "
Oct 31 18:46:17.171506 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.171450    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f80435b-76fb-41c8-a6a6-c5d99fe23021-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:17.183092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.183064    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-maas-api-tls" pod "5f80435b-76fb-41c8-a6a6-c5d99fe23021" (UID: "5f80435b-76fb-41c8-a6a6-c5d99fe23021"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:46:17.183316 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.183285    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f80435b-76fb-41c8-a6a6-c5d99fe23021-kube-api-access-6cm6p" pod "5f80435b-76fb-41c8-a6a6-c5d99fe23021" (UID: "5f80435b-76fb-41c8-a6a6-c5d99fe23021"). InnerVolumeSpecName "kube-api-access-6cm6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:46:17.183442 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.183372    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-metrics-certs" pod "5f80435b-76fb-41c8-a6a6-c5d99fe23021" (UID: "5f80435b-76fb-41c8-a6a6-c5d99fe23021"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:46:17.273199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.273109    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:17.273199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.273145    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f80435b-76fb-41c8-a6a6-c5d99fe23021-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:17.273199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.273162    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cm6p\" (UniqueName: \"kubernetes.io/projected/5f80435b-76fb-41c8-a6a6-c5d99fe23021-kube-api-access-6cm6p\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:17.852254 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.852221    4484 generic.go:356] "Generic (PLEG): container finished" podID="5f80435b-76fb-41c8-a6a6-c5d99fe23021" containerID="5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76" exitCode=0
Oct 31 18:46:17.852706 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.852273    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"
Oct 31 18:46:17.852706 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.852288    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp" event={"ID":"5f80435b-76fb-41c8-a6a6-c5d99fe23021","Type":"ContainerDied","Data":"5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76"}
Oct 31 18:46:17.852706 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.852331    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp" event={"ID":"5f80435b-76fb-41c8-a6a6-c5d99fe23021","Type":"ContainerDied","Data":"5292009ebe9f425400bed5f09bf335c261da89a011595fe4916ccea5ab60e0c5"}
Oct 31 18:46:17.852706 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.852351    4484 scope.go:122] "RemoveContainer" containerID="5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76"
Oct 31 18:46:17.864744 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.864724    4484 scope.go:122] "RemoveContainer" containerID="5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76"
Oct 31 18:46:17.865082 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:46:17.865061    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76\": container with ID starting with 5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76 not found: ID does not exist" containerID="5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76"
Oct 31 18:46:17.865149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:17.865092    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76"} err="failed to get container status \"5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76\": rpc error: code = NotFound desc = could not find container \"5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76\": container with ID starting with 5b39433be52878b22af0ad2859b057dfee83065f699c3afa203b507633cd3b76 not found: ID does not exist"
Oct 31 18:46:18.019784 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:18.019746    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"]
Oct 31 18:46:18.070120 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:18.070089    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bd6c0a-6d6b968678-hkwzp"]
Oct 31 18:46:18.688169 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:18.688139    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:19.828732 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:19.828692    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:20.005243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:20.005213    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f80435b-76fb-41c8-a6a6-c5d99fe23021" path="/var/lib/kubelet/pods/5f80435b-76fb-41c8-a6a6-c5d99fe23021/volumes"
Oct 31 18:46:20.037056 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:20.037031    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:21.425226 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:21.425190    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:21.957931 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:21.957896    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"]
Oct 31 18:46:21.958155 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:21.958131    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerName="istio-proxy" containerID="cri-o://9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e" gracePeriod=30
Oct 31 18:46:21.972727 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:21.972697    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:22.855890 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:22.855858    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:23.713221 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:23.713184    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:25.468200 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:46:25.468154    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:46:27.241845 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.241816    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:46:27.316369 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.316337    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-credential-socket\") pod \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") "
Oct 31 18:46:27.316534 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.316410    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-certs\") pod \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") "
Oct 31 18:46:27.316534 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.316466    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istiod-ca-cert\") pod \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") "
Oct 31 18:46:27.316661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.316578    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-kube-api-access-2mkhn\" (UniqueName: \"kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-kube-api-access-2mkhn\") pod \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") "
Oct 31 18:46:27.316661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.316611    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-credential-socket" pod "e017eb31-06e0-4b5d-8414-3df2bd85ecab" (UID: "e017eb31-06e0-4b5d-8414-3df2bd85ecab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:46:27.316661 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.316624    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-token\" (UniqueName: \"kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-token\") pod \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") "
Oct 31 18:46:27.316818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.316729    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-envoy\") pod \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") "
Oct 31 18:46:27.316818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.316744    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-certs" pod "e017eb31-06e0-4b5d-8414-3df2bd85ecab" (UID: "e017eb31-06e0-4b5d-8414-3df2bd85ecab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:46:27.316922 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.316850    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-socket\") pod \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") "
Oct 31 18:46:27.316922 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.316890    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-podinfo\") pod \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") "
Oct 31 18:46:27.316922 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.316911    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istiod-ca-cert" pod "e017eb31-06e0-4b5d-8414-3df2bd85ecab" (UID: "e017eb31-06e0-4b5d-8414-3df2bd85ecab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:46:27.317071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.316969    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-data\") pod \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\" (UID: \"e017eb31-06e0-4b5d-8414-3df2bd85ecab\") "
Oct 31 18:46:27.317187 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.317146    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-socket" pod "e017eb31-06e0-4b5d-8414-3df2bd85ecab" (UID: "e017eb31-06e0-4b5d-8414-3df2bd85ecab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:46:27.317557 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.317531    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-data" pod "e017eb31-06e0-4b5d-8414-3df2bd85ecab" (UID: "e017eb31-06e0-4b5d-8414-3df2bd85ecab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:46:27.317695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.317563    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:27.317695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.317582    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:27.317695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.317595    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:27.317695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.317609    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:27.319087 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.319058    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-envoy" pod "e017eb31-06e0-4b5d-8414-3df2bd85ecab" (UID: "e017eb31-06e0-4b5d-8414-3df2bd85ecab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:46:27.319173 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.319142    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-podinfo" pod "e017eb31-06e0-4b5d-8414-3df2bd85ecab" (UID: "e017eb31-06e0-4b5d-8414-3df2bd85ecab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:46:27.319391 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.319374    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-token" pod "e017eb31-06e0-4b5d-8414-3df2bd85ecab" (UID: "e017eb31-06e0-4b5d-8414-3df2bd85ecab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:46:27.319450 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.319388    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-kube-api-access-2mkhn" pod "e017eb31-06e0-4b5d-8414-3df2bd85ecab" (UID: "e017eb31-06e0-4b5d-8414-3df2bd85ecab"). InnerVolumeSpecName "kube-api-access-2mkhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:46:27.419136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.419103    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mkhn\" (UniqueName: \"kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-kube-api-access-2mkhn\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:27.419136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.419134    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:27.419321 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.419146    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:27.419321 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.419160    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:27.419321 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.419173    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e017eb31-06e0-4b5d-8414-3df2bd85ecab-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:27.924048 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.924014    4484 generic.go:356] "Generic (PLEG): container finished" podID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerID="9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e" exitCode=0
Oct 31 18:46:27.924246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.924068    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"
Oct 31 18:46:27.924246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.924099    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" event={"ID":"e017eb31-06e0-4b5d-8414-3df2bd85ecab","Type":"ContainerDied","Data":"9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e"}
Oct 31 18:46:27.924246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.924175    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq" event={"ID":"e017eb31-06e0-4b5d-8414-3df2bd85ecab","Type":"ContainerDied","Data":"f6dde50937aa7b1cb7fc3cdf8cf6598560be16212eb727b196e127ba5dfea67d"}
Oct 31 18:46:27.924246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.924202    4484 scope.go:122] "RemoveContainer" containerID="9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e"
Oct 31 18:46:27.934838 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.934820    4484 scope.go:122] "RemoveContainer" containerID="c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a"
Oct 31 18:46:27.944746 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.944726    4484 scope.go:122] "RemoveContainer" containerID="9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e"
Oct 31 18:46:27.944998 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:46:27.944977    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e\": container with ID starting with 9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e not found: ID does not exist" containerID="9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e"
Oct 31 18:46:27.945062 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.945005    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e"} err="failed to get container status \"9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e\": rpc error: code = NotFound desc = could not find container \"9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e\": container with ID starting with 9a07e85780edb867b9a6037d87e22f8bc1f4f44888db4a29779de77135e9832e not found: ID does not exist"
Oct 31 18:46:27.945062 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.945021    4484 scope.go:122] "RemoveContainer" containerID="c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a"
Oct 31 18:46:27.945299 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:46:27.945270    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a\": container with ID starting with c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a not found: ID does not exist" containerID="c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a"
Oct 31 18:46:27.945348 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:27.945305    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a"} err="failed to get container status \"c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a\": rpc error: code = NotFound desc = could not find container \"c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a\": container with ID starting with c5008f0c4059306717f542864e09180d7e9b18de339f659d7e85dc37c3b0249a not found: ID does not exist"
Oct 31 18:46:28.143617 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:28.143586    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"]
Oct 31 18:46:28.192059 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:28.191982    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bd6c0a-openshift-default-6f7c7bdcbf-g59bq"]
Oct 31 18:46:28.710032 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:28.709994    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:28.867075 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:28.867041    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:29.191197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:29.191166    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"]
Oct 31 18:46:29.254354 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:29.254314    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:46:29.538574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:29.538538    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:29.697669 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:29.697635    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:30.004887 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:30.004852    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" path="/var/lib/kubelet/pods/e017eb31-06e0-4b5d-8414-3df2bd85ecab/volumes"
Oct 31 18:46:33.877282 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:33.877250    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:34.192277 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.192194    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl" podUID="93d8865e-abfc-4a75-8b38-e5458b5eb570" containerName="payload-processing" containerID="cri-o://9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4" gracePeriod=25
Oct 31 18:46:34.449599 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.449543    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"
Oct 31 18:46:34.620636 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.620602    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d8865e-abfc-4a75-8b38-e5458b5eb570-kube-api-access-bvz46\" (UniqueName: \"kubernetes.io/projected/93d8865e-abfc-4a75-8b38-e5458b5eb570-kube-api-access-bvz46\") pod \"93d8865e-abfc-4a75-8b38-e5458b5eb570\" (UID: \"93d8865e-abfc-4a75-8b38-e5458b5eb570\") "
Oct 31 18:46:34.620813 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.620679    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d8865e-abfc-4a75-8b38-e5458b5eb570-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d8865e-abfc-4a75-8b38-e5458b5eb570-plugins-config-volume\") pod \"93d8865e-abfc-4a75-8b38-e5458b5eb570\" (UID: \"93d8865e-abfc-4a75-8b38-e5458b5eb570\") "
Oct 31 18:46:34.621069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.621046    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d8865e-abfc-4a75-8b38-e5458b5eb570-plugins-config-volume" pod "93d8865e-abfc-4a75-8b38-e5458b5eb570" (UID: "93d8865e-abfc-4a75-8b38-e5458b5eb570"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:46:34.621306 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.621254    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d8865e-abfc-4a75-8b38-e5458b5eb570-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:34.622717 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.622696    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d8865e-abfc-4a75-8b38-e5458b5eb570-kube-api-access-bvz46" pod "93d8865e-abfc-4a75-8b38-e5458b5eb570" (UID: "93d8865e-abfc-4a75-8b38-e5458b5eb570"). InnerVolumeSpecName "kube-api-access-bvz46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:46:34.678342 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.678312    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:34.722838 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.722777    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvz46\" (UniqueName: \"kubernetes.io/projected/93d8865e-abfc-4a75-8b38-e5458b5eb570-kube-api-access-bvz46\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:46:34.979654 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.979570    4484 generic.go:356] "Generic (PLEG): container finished" podID="93d8865e-abfc-4a75-8b38-e5458b5eb570" containerID="9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4" exitCode=0
Oct 31 18:46:34.979654 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.979613    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"
Oct 31 18:46:34.980157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.979653    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl" event={"ID":"93d8865e-abfc-4a75-8b38-e5458b5eb570","Type":"ContainerDied","Data":"9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4"}
Oct 31 18:46:34.980157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.979711    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl" event={"ID":"93d8865e-abfc-4a75-8b38-e5458b5eb570","Type":"ContainerDied","Data":"2f43d44013b18e2395aec1657354d5ca611e996d648bb4abda700feabfa6cdaf"}
Oct 31 18:46:34.980157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.979746    4484 scope.go:122] "RemoveContainer" containerID="9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4"
Oct 31 18:46:34.989720 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.989703    4484 scope.go:122] "RemoveContainer" containerID="9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4"
Oct 31 18:46:34.990008 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:46:34.989986    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4\": container with ID starting with 9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4 not found: ID does not exist" containerID="9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4"
Oct 31 18:46:34.990077 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:34.990020    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4"} err="failed to get container status \"9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4\": rpc error: code = NotFound desc = could not find container \"9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4\": container with ID starting with 9ff00a862a6517924931819491ebef32d98ada400ab3b8089fe8e5cddd6b98e4 not found: ID does not exist"
Oct 31 18:46:35.144753 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:35.144719    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"]
Oct 31 18:46:35.195083 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:35.195052    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-373109-7ccdd56999-rhxcl"]
Oct 31 18:46:35.994224 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:35.994184    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d8865e-abfc-4a75-8b38-e5458b5eb570" path="/var/lib/kubelet/pods/93d8865e-abfc-4a75-8b38-e5458b5eb570/volumes"
Oct 31 18:46:39.000687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:39.000655    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:44.128508 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:44.128445    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:48.645801 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:48.645767    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:48.861568 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:48.861432    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:49.914638 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:49.914600    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:50.397724 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:50.397686    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:50.678263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:50.678162    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:50.835156 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:50.835124    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:50.871253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:50.871216    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"]
Oct 31 18:46:50.920993 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:50.920955    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3417b822-x86lr"]
Oct 31 18:46:51.114099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:51.114062    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:51.995182 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:51.995150    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a1bd12-cbc4-43af-bae3-cb29092bf68f" path="/var/lib/kubelet/pods/c0a1bd12-cbc4-43af-bae3-cb29092bf68f/volumes"
Oct 31 18:46:53.451592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:53.451553    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:54.399685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:54.399658    4484 scope.go:122] "RemoveContainer" containerID="95415b36551f7c310225222dedcc26c2200f62343b161fc1dc09473afc546ee7"
Oct 31 18:46:54.409424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:54.409401    4484 scope.go:122] "RemoveContainer" containerID="b7f5bd7c70a24a496280bafd4bc8fcf979f1790c43c6a24fb6ed01b65929eb59"
Oct 31 18:46:54.418329 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:54.418308    4484 scope.go:122] "RemoveContainer" containerID="34c456b8de3e4c00faaa2160ee3792f6879e799e3a3be84b20d345ea7b768e19"
Oct 31 18:46:54.427103 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:54.427086    4484 scope.go:122] "RemoveContainer" containerID="e832cd2b1ebfcf039df34944bd83322c7683a99fcb0fa9212025d9503ed4986b"
Oct 31 18:46:54.831234 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:54.831204    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:55.467671 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:46:55.467632    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:46:55.832060 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:55.832021    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:57.275429 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:57.275389    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:57.662926 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:57.662842    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"]
Oct 31 18:46:57.774717 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:57.774656    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"]
Oct 31 18:46:58.038069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:58.038040    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:58.235092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:58.235058    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:58.635851 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:58.635816    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:46:58.790741 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:46:58.790709    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:01.004309 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.004264    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:01.425053 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.424963    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:01.605444 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.605409    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:01.820134 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820104    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"]
Oct 31 18:47:01.820800 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820786    4484 cpu_manager.go:414] "containerMap: removing container" podUID="97aced5d-a853-4858-bbe7-fbe1088dd7fe" containerName="payload-pre-processing"
Oct 31 18:47:01.820858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820801    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97aced5d-a853-4858-bbe7-fbe1088dd7fe" containerName="payload-pre-processing"
Oct 31 18:47:01.820858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820815    4484 cpu_manager.go:414] "containerMap: removing container" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerName="istio-proxy"
Oct 31 18:47:01.820858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820820    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerName="istio-proxy"
Oct 31 18:47:01.820858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820831    4484 cpu_manager.go:414] "containerMap: removing container" podUID="93d8865e-abfc-4a75-8b38-e5458b5eb570" containerName="payload-processing"
Oct 31 18:47:01.820858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820835    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d8865e-abfc-4a75-8b38-e5458b5eb570" containerName="payload-processing"
Oct 31 18:47:01.820858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820851    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5f80435b-76fb-41c8-a6a6-c5d99fe23021" containerName="maas-api"
Oct 31 18:47:01.820858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820856    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f80435b-76fb-41c8-a6a6-c5d99fe23021" containerName="maas-api"
Oct 31 18:47:01.821071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820963    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="97aced5d-a853-4858-bbe7-fbe1088dd7fe" containerName="payload-pre-processing"
Oct 31 18:47:01.821071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820971    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerName="istio-proxy"
Oct 31 18:47:01.821071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820976    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d8865e-abfc-4a75-8b38-e5458b5eb570" containerName="payload-processing"
Oct 31 18:47:01.821071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820985    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f80435b-76fb-41c8-a6a6-c5d99fe23021" containerName="maas-api"
Oct 31 18:47:01.821071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.820993    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerName="istio-proxy"
Oct 31 18:47:01.826254 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.826236    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"
Oct 31 18:47:01.882183 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.882155    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"]
Oct 31 18:47:01.936918 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.936886    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3417b822-dockercfg-bpzd4\""
Oct 31 18:47:01.956824 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:01.937229    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3417b822\""
Oct 31 18:47:02.008177 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.008134    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"]
Oct 31 18:47:02.011545 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.011519    4484 cpu_manager.go:414] "containerMap: removing container" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerName="istio-proxy"
Oct 31 18:47:02.012199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.012180    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e017eb31-06e0-4b5d-8414-3df2bd85ecab" containerName="istio-proxy"
Oct 31 18:47:02.014840 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.014790    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps52d\" (UniqueName: \"kubernetes.io/projected/cbdb1b51-def9-40b6-818c-3009e3969017-kube-api-access-ps52d\") pod \"payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8\" (UID: \"cbdb1b51-def9-40b6-818c-3009e3969017\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"
Oct 31 18:47:02.015076 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.015055    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdb1b51-def9-40b6-818c-3009e3969017-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8\" (UID: \"cbdb1b51-def9-40b6-818c-3009e3969017\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"
Oct 31 18:47:02.028522 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.021645    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"
Oct 31 18:47:02.049654 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.043889    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"]
Oct 31 18:47:02.116224 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.116148    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81333aa2-5037-4dd9-8fce-3f1cf2531072-plugins-config-volume\") pod \"payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9\" (UID: \"81333aa2-5037-4dd9-8fce-3f1cf2531072\") " pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"
Oct 31 18:47:02.116224 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.116196    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cddvp\" (UniqueName: \"kubernetes.io/projected/81333aa2-5037-4dd9-8fce-3f1cf2531072-kube-api-access-cddvp\") pod \"payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9\" (UID: \"81333aa2-5037-4dd9-8fce-3f1cf2531072\") " pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"
Oct 31 18:47:02.116451 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.116401    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdb1b51-def9-40b6-818c-3009e3969017-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8\" (UID: \"cbdb1b51-def9-40b6-818c-3009e3969017\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"
Oct 31 18:47:02.116643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.116626    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps52d\" (UniqueName: \"kubernetes.io/projected/cbdb1b51-def9-40b6-818c-3009e3969017-kube-api-access-ps52d\") pod \"payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8\" (UID: \"cbdb1b51-def9-40b6-818c-3009e3969017\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"
Oct 31 18:47:02.117125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.117102    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdb1b51-def9-40b6-818c-3009e3969017-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8\" (UID: \"cbdb1b51-def9-40b6-818c-3009e3969017\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"
Oct 31 18:47:02.174869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.174833    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps52d\" (UniqueName: \"kubernetes.io/projected/cbdb1b51-def9-40b6-818c-3009e3969017-kube-api-access-ps52d\") pod \"payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8\" (UID: \"cbdb1b51-def9-40b6-818c-3009e3969017\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"
Oct 31 18:47:02.218013 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.217988    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81333aa2-5037-4dd9-8fce-3f1cf2531072-plugins-config-volume\") pod \"payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9\" (UID: \"81333aa2-5037-4dd9-8fce-3f1cf2531072\") " pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"
Oct 31 18:47:02.218189 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.218030    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cddvp\" (UniqueName: \"kubernetes.io/projected/81333aa2-5037-4dd9-8fce-3f1cf2531072-kube-api-access-cddvp\") pod \"payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9\" (UID: \"81333aa2-5037-4dd9-8fce-3f1cf2531072\") " pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"
Oct 31 18:47:02.218726 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.218703    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81333aa2-5037-4dd9-8fce-3f1cf2531072-plugins-config-volume\") pod \"payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9\" (UID: \"81333aa2-5037-4dd9-8fce-3f1cf2531072\") " pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"
Oct 31 18:47:02.276219 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.276194    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cddvp\" (UniqueName: \"kubernetes.io/projected/81333aa2-5037-4dd9-8fce-3f1cf2531072-kube-api-access-cddvp\") pod \"payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9\" (UID: \"81333aa2-5037-4dd9-8fce-3f1cf2531072\") " pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"
Oct 31 18:47:02.345174 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.345137    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"
Oct 31 18:47:02.440702 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.440665    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"
Oct 31 18:47:02.663804 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.663715    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s" podUID="88244ad4-1418-4590-8e3d-f7ab7222482f" containerName="payload-processing" containerID="cri-o://75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2" gracePeriod=25
Oct 31 18:47:02.721715 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.721690    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"]
Oct 31 18:47:02.731452 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:47:02.731422    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81333aa2_5037_4dd9_8fce_3f1cf2531072.slice/crio-e0934e4d77128bf51486393cf10c376e269ad6a3ade89b42162e78f49b9c8d2b WatchSource:0}: Error finding container e0934e4d77128bf51486393cf10c376e269ad6a3ade89b42162e78f49b9c8d2b: Status 404 returned error can't find the container with id e0934e4d77128bf51486393cf10c376e269ad6a3ade89b42162e78f49b9c8d2b
Oct 31 18:47:02.743812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.743783    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:02.775919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.775893    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j" podUID="cdfc1774-5d73-4c87-8c82-141edcb3a067" containerName="payload-pre-processing" containerID="cri-o://8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8" gracePeriod=25
Oct 31 18:47:02.820246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.820224    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"]
Oct 31 18:47:02.848172 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:47:02.846859    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdb1b51_def9_40b6_818c_3009e3969017.slice/crio-d7b9d77f5e35ac867cff1a88f0536ab2dc925c61681d99b504bc07ae123aca4c WatchSource:0}: Error finding container d7b9d77f5e35ac867cff1a88f0536ab2dc925c61681d99b504bc07ae123aca4c: Status 404 returned error can't find the container with id d7b9d77f5e35ac867cff1a88f0536ab2dc925c61681d99b504bc07ae123aca4c
Oct 31 18:47:02.931888 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:02.931855    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"
Oct 31 18:47:03.028305 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.028268    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88244ad4-1418-4590-8e3d-f7ab7222482f-kube-api-access-cvtxf\" (UniqueName: \"kubernetes.io/projected/88244ad4-1418-4590-8e3d-f7ab7222482f-kube-api-access-cvtxf\") pod \"88244ad4-1418-4590-8e3d-f7ab7222482f\" (UID: \"88244ad4-1418-4590-8e3d-f7ab7222482f\") "
Oct 31 18:47:03.028812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.028492    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88244ad4-1418-4590-8e3d-f7ab7222482f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88244ad4-1418-4590-8e3d-f7ab7222482f-plugins-config-volume\") pod \"88244ad4-1418-4590-8e3d-f7ab7222482f\" (UID: \"88244ad4-1418-4590-8e3d-f7ab7222482f\") "
Oct 31 18:47:03.029548 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.029521    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88244ad4-1418-4590-8e3d-f7ab7222482f-plugins-config-volume" pod "88244ad4-1418-4590-8e3d-f7ab7222482f" (UID: "88244ad4-1418-4590-8e3d-f7ab7222482f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:03.032013 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.031980    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88244ad4-1418-4590-8e3d-f7ab7222482f-kube-api-access-cvtxf" pod "88244ad4-1418-4590-8e3d-f7ab7222482f" (UID: "88244ad4-1418-4590-8e3d-f7ab7222482f"). InnerVolumeSpecName "kube-api-access-cvtxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:03.065390 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.065366    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"
Oct 31 18:47:03.129758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.129731    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdfc1774-5d73-4c87-8c82-141edcb3a067-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdfc1774-5d73-4c87-8c82-141edcb3a067-plugins-config-volume\") pod \"cdfc1774-5d73-4c87-8c82-141edcb3a067\" (UID: \"cdfc1774-5d73-4c87-8c82-141edcb3a067\") "
Oct 31 18:47:03.129920 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.129773    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdfc1774-5d73-4c87-8c82-141edcb3a067-kube-api-access-96f8b\" (UniqueName: \"kubernetes.io/projected/cdfc1774-5d73-4c87-8c82-141edcb3a067-kube-api-access-96f8b\") pod \"cdfc1774-5d73-4c87-8c82-141edcb3a067\" (UID: \"cdfc1774-5d73-4c87-8c82-141edcb3a067\") "
Oct 31 18:47:03.130161 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.130132    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdfc1774-5d73-4c87-8c82-141edcb3a067-plugins-config-volume" pod "cdfc1774-5d73-4c87-8c82-141edcb3a067" (UID: "cdfc1774-5d73-4c87-8c82-141edcb3a067"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:03.130600 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.130555    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88244ad4-1418-4590-8e3d-f7ab7222482f-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:03.130600 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.130576    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdfc1774-5d73-4c87-8c82-141edcb3a067-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:03.130600 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.130587    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvtxf\" (UniqueName: \"kubernetes.io/projected/88244ad4-1418-4590-8e3d-f7ab7222482f-kube-api-access-cvtxf\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:03.132000 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.131978    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdfc1774-5d73-4c87-8c82-141edcb3a067-kube-api-access-96f8b" pod "cdfc1774-5d73-4c87-8c82-141edcb3a067" (UID: "cdfc1774-5d73-4c87-8c82-141edcb3a067"). InnerVolumeSpecName "kube-api-access-96f8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:03.170136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.170035    4484 generic.go:356] "Generic (PLEG): container finished" podID="88244ad4-1418-4590-8e3d-f7ab7222482f" containerID="75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2" exitCode=0
Oct 31 18:47:03.170136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.170082    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"
Oct 31 18:47:03.170136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.170122    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s" event={"ID":"88244ad4-1418-4590-8e3d-f7ab7222482f","Type":"ContainerDied","Data":"75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2"}
Oct 31 18:47:03.170348 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.170186    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s" event={"ID":"88244ad4-1418-4590-8e3d-f7ab7222482f","Type":"ContainerDied","Data":"d04e04474fda82f37d111fd5036c3bc3e44734fba5ec1a7cd0d2173206190cda"}
Oct 31 18:47:03.170348 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.170214    4484 scope.go:122] "RemoveContainer" containerID="75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2"
Oct 31 18:47:03.171843 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.171820    4484 generic.go:356] "Generic (PLEG): container finished" podID="cdfc1774-5d73-4c87-8c82-141edcb3a067" containerID="8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8" exitCode=0
Oct 31 18:47:03.171951 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.171863    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"
Oct 31 18:47:03.171951 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.171898    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j" event={"ID":"cdfc1774-5d73-4c87-8c82-141edcb3a067","Type":"ContainerDied","Data":"8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8"}
Oct 31 18:47:03.172041 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.171958    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j" event={"ID":"cdfc1774-5d73-4c87-8c82-141edcb3a067","Type":"ContainerDied","Data":"886ae47bf1dd8318bf9d8fd1c749b1a537e661fc2c08106fb73ef3c7256f7a66"}
Oct 31 18:47:03.173912 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.173886    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8" event={"ID":"cbdb1b51-def9-40b6-818c-3009e3969017","Type":"ContainerStarted","Data":"e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c"}
Oct 31 18:47:03.174037 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.173926    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8" event={"ID":"cbdb1b51-def9-40b6-818c-3009e3969017","Type":"ContainerStarted","Data":"d7b9d77f5e35ac867cff1a88f0536ab2dc925c61681d99b504bc07ae123aca4c"}
Oct 31 18:47:03.174037 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.173977    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"
Oct 31 18:47:03.176396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.176372    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9" event={"ID":"81333aa2-5037-4dd9-8fce-3f1cf2531072","Type":"ContainerStarted","Data":"f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83"}
Oct 31 18:47:03.176549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.176423    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"
Oct 31 18:47:03.176549 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.176447    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9" event={"ID":"81333aa2-5037-4dd9-8fce-3f1cf2531072","Type":"ContainerStarted","Data":"e0934e4d77128bf51486393cf10c376e269ad6a3ade89b42162e78f49b9c8d2b"}
Oct 31 18:47:03.210936 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.210912    4484 scope.go:122] "RemoveContainer" containerID="75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2"
Oct 31 18:47:03.211372 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:47:03.211340    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2\": container with ID starting with 75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2 not found: ID does not exist" containerID="75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2"
Oct 31 18:47:03.211507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.211384    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2"} err="failed to get container status \"75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2\": rpc error: code = NotFound desc = could not find container \"75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2\": container with ID starting with 75e1cd031474870fa2882461b47df019bd3dae0a828f5e9df687e53b927ccac2 not found: ID does not exist"
Oct 31 18:47:03.211507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.211407    4484 scope.go:122] "RemoveContainer" containerID="8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8"
Oct 31 18:47:03.221406 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.221384    4484 scope.go:122] "RemoveContainer" containerID="8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8"
Oct 31 18:47:03.221736 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:47:03.221711    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8\": container with ID starting with 8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8 not found: ID does not exist" containerID="8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8"
Oct 31 18:47:03.221820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.221748    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8"} err="failed to get container status \"8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8\": rpc error: code = NotFound desc = could not find container \"8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8\": container with ID starting with 8c650a02a6b628b6d6521be4e2b27ce0000018822807de41ae4134c82cb938c8 not found: ID does not exist"
Oct 31 18:47:03.231736 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.231716    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96f8b\" (UniqueName: \"kubernetes.io/projected/cdfc1774-5d73-4c87-8c82-141edcb3a067-kube-api-access-96f8b\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:03.401375 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.401323    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8" podStartSLOduration=2.401306445 podStartE2EDuration="2.401306445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:47:03.400255162 +0000 UTC m=+1569.546705026" watchObservedRunningTime="2026-10-31 18:47:03.401306445 +0000 UTC m=+1569.547756303"
Oct 31 18:47:03.514580 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.514520    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9" podStartSLOduration=2.514459679 podStartE2EDuration="2.514459679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:47:03.513447693 +0000 UTC m=+1569.659897559" watchObservedRunningTime="2026-10-31 18:47:03.514459679 +0000 UTC m=+1569.660909546"
Oct 31 18:47:03.627000 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.626955    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"]
Oct 31 18:47:03.631084 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.631064    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a6c8da-5c75f8657-mc56s"]
Oct 31 18:47:03.751792 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.748492    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"]
Oct 31 18:47:03.754278 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.754072    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a6c8da-98476b467-t2g4j"]
Oct 31 18:47:03.902196 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.902110    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:47:03.995776 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.995745    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88244ad4-1418-4590-8e3d-f7ab7222482f" path="/var/lib/kubelet/pods/88244ad4-1418-4590-8e3d-f7ab7222482f/volumes"
Oct 31 18:47:03.996135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:03.996121    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdfc1774-5d73-4c87-8c82-141edcb3a067" path="/var/lib/kubelet/pods/cdfc1774-5d73-4c87-8c82-141edcb3a067/volumes"
Oct 31 18:47:06.005996 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:06.005925    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"]
Oct 31 18:47:06.006507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:06.006103    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" podUID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" containerName="istio-proxy" containerID="cri-o://0a06e241445c11c5c98dd80787ae60f68668d565e813a1db0cb78e1b75e63394" gracePeriod=30
Oct 31 18:47:06.056460 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:06.056431    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"]
Oct 31 18:47:06.056665 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:06.056645    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp" podUID="d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" containerName="istio-proxy" containerID="cri-o://231b15f359fd930d52831d606f71f575e31459c1619462414fad41aca854b729" gracePeriod=30
Oct 31 18:47:06.309396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:06.309313    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:10.095634 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:10.095599    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:10.830872 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:10.830837    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:11.246331 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.246297    4484 generic.go:356] "Generic (PLEG): container finished" podID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" containerID="0a06e241445c11c5c98dd80787ae60f68668d565e813a1db0cb78e1b75e63394" exitCode=0
Oct 31 18:47:11.246867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.246408    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" event={"ID":"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f","Type":"ContainerDied","Data":"0a06e241445c11c5c98dd80787ae60f68668d565e813a1db0cb78e1b75e63394"}
Oct 31 18:47:11.249059 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.248944    4484 generic.go:356] "Generic (PLEG): container finished" podID="d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" containerID="231b15f359fd930d52831d606f71f575e31459c1619462414fad41aca854b729" exitCode=0
Oct 31 18:47:11.249267 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.249212    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp" event={"ID":"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f","Type":"ContainerDied","Data":"231b15f359fd930d52831d606f71f575e31459c1619462414fad41aca854b729"}
Oct 31 18:47:11.410881 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.410419    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:47:11.418296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.418096    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"]
Oct 31 18:47:11.418764 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.418747    4484 cpu_manager.go:414] "containerMap: removing container" podUID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" containerName="istio-proxy"
Oct 31 18:47:11.418764 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.418763    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" containerName="istio-proxy"
Oct 31 18:47:11.418901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.418799    4484 cpu_manager.go:414] "containerMap: removing container" podUID="88244ad4-1418-4590-8e3d-f7ab7222482f" containerName="payload-processing"
Oct 31 18:47:11.418901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.418805    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88244ad4-1418-4590-8e3d-f7ab7222482f" containerName="payload-processing"
Oct 31 18:47:11.418901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.418821    4484 cpu_manager.go:414] "containerMap: removing container" podUID="cdfc1774-5d73-4c87-8c82-141edcb3a067" containerName="payload-pre-processing"
Oct 31 18:47:11.418901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.418826    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdfc1774-5d73-4c87-8c82-141edcb3a067" containerName="payload-pre-processing"
Oct 31 18:47:11.419094 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.418914    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" containerName="istio-proxy"
Oct 31 18:47:11.419094 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.418926    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="88244ad4-1418-4590-8e3d-f7ab7222482f" containerName="payload-processing"
Oct 31 18:47:11.419094 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.418934    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdfc1774-5d73-4c87-8c82-141edcb3a067" containerName="payload-pre-processing"
Oct 31 18:47:11.422671 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.422650    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.439307 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.439286    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:47:11.474578 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.474552    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"]
Oct 31 18:47:11.494794 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.494754    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:11.513901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.513874    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-66032c-openshift-default-dockercfg-vgf4r\""
Oct 31 18:47:11.547118 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.547084    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-kube-api-access-wpsdw\" (UniqueName: \"kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-kube-api-access-wpsdw\") pod \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") "
Oct 31 18:47:11.547422 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.547399    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-envoy\") pod \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") "
Oct 31 18:47:11.547663 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.547646    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-credential-socket\") pod \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") "
Oct 31 18:47:11.548522 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.548144    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istiod-ca-cert\") pod \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") "
Oct 31 18:47:11.548622 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.548075    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-credential-socket" pod "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" (UID: "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:11.548622 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.548460    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istiod-ca-cert" pod "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" (UID: "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:11.548622 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.548618    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-data\") pod \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") "
Oct 31 18:47:11.548916 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.548896    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-data" pod "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" (UID: "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:11.548982 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.548954    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-token\" (UniqueName: \"kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-token\") pod \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") "
Oct 31 18:47:11.549017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.549006    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-socket\") pod \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") "
Oct 31 18:47:11.549125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.549110    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-podinfo\") pod \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") "
Oct 31 18:47:11.549195 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.549146    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-podinfo\") pod \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") "
Oct 31 18:47:11.549871 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.549610    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-certs\") pod \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") "
Oct 31 18:47:11.549975 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.549893    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-certs" pod "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" (UID: "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:11.549975 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.549936    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-credential-socket\") pod \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") "
Oct 31 18:47:11.550176 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.550142    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-credential-socket" pod "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" (UID: "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:11.550285 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.550256    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-kube-api-access-wpsdw" pod "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" (UID: "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f"). InnerVolumeSpecName "kube-api-access-wpsdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:11.550451 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.550433    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-socket\") pod \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") "
Oct 31 18:47:11.550890 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.550464    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-socket" pod "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" (UID: "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:11.550890 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.550762    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-socket" pod "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" (UID: "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:11.551094 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.551074    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-data\") pod \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") "
Oct 31 18:47:11.551491 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.551367    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-data" pod "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" (UID: "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:11.551639 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.551623    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-kube-api-access-mlx22\" (UniqueName: \"kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-kube-api-access-mlx22\") pod \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") "
Oct 31 18:47:11.552126 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.552108    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-envoy\") pod \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") "
Oct 31 18:47:11.552335 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.552317    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-token\" (UniqueName: \"kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-token\") pod \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") "
Oct 31 18:47:11.552466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.552452    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istiod-ca-cert\") pod \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\" (UID: \"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f\") "
Oct 31 18:47:11.553345 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.552891    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istiod-ca-cert" pod "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" (UID: "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:11.553748 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.553717    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-certs" pod "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" (UID: "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:11.553877 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.553859    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-certs\") pod \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\" (UID: \"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f\") "
Oct 31 18:47:11.554199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.554177    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx52w\" (UniqueName: \"kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-kube-api-access-mx52w\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.554428 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.554408    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-socket\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.554658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.554638    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-envoy\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.554858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.554718    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-envoy" pod "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" (UID: "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:11.554939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.554918    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-kube-api-access-mlx22" pod "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" (UID: "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f"). InnerVolumeSpecName "kube-api-access-mlx22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:11.555153 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555130    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-data\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.555252 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555225    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-istio-token\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.555311 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555252    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0159e4a4-ad64-49f0-9256-9922e333235e-istiod-ca-cert\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.555419 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555401    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-token" pod "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" (UID: "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:11.555521 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555504    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-certs\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.555575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555565    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0159e4a4-ad64-49f0-9256-9922e333235e-istio-podinfo\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.555636 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555599    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-credential-socket\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.555731 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555713    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.555830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555733    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.555830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555747    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.555830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555762    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpsdw\" (UniqueName: \"kubernetes.io/projected/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-kube-api-access-wpsdw\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.555830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555776    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.555830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555797    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.555830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555805    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.555830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555814    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.555830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555823    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.555830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555831    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.556276 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555840    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.556276 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555847    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.556276 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555858    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlx22\" (UniqueName: \"kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-kube-api-access-mlx22\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.556276 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.555871    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.557879 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.557843    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-podinfo" pod "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" (UID: "d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:47:11.557879 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.557857    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-token" pod "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" (UID: "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:11.558030 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.557986    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-envoy" pod "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" (UID: "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:11.559077 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.559054    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-podinfo" pod "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" (UID: "7fc48784-e2e5-4d2f-90f0-9edbe9c9644f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:47:11.657193 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657111    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-data\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.657193 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657184    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-istio-token\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.657485 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657210    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0159e4a4-ad64-49f0-9256-9922e333235e-istiod-ca-cert\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.657485 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657317    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-certs\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.657485 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657365    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0159e4a4-ad64-49f0-9256-9922e333235e-istio-podinfo\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.657485 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657391    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-credential-socket\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.657485 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657456    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx52w\" (UniqueName: \"kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-kube-api-access-mx52w\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.657738 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657522    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-socket\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.657738 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657531    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-data\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.657738 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657559    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-envoy\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.657738 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657644    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.657738 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657659    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.657738 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657673    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.657738 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.657687    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:11.658576 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.658379    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-credential-socket\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.659845 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.658992    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0159e4a4-ad64-49f0-9256-9922e333235e-istiod-ca-cert\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.659845 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.659629    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-certs\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.659845 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.659640    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-socket\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.660118 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.660098    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-envoy\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.661516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.661456    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0159e4a4-ad64-49f0-9256-9922e333235e-istio-podinfo\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.710766 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.710723    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:11.714951 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.714923    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx52w\" (UniqueName: \"kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-kube-api-access-mx52w\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.715821 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.715801    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-istio-token\") pod \"e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") " pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.749830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.749809    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:11.960024 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:11.959898    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:12.200786 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.200753    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"]
Oct 31 18:47:12.201877 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:47:12.201839    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0159e4a4_ad64_49f0_9256_9922e333235e.slice/crio-bf959b3d6a129769748d11dc9ca956f2035a9352565079dc7b817f51981516ce WatchSource:0}: Error finding container bf959b3d6a129769748d11dc9ca956f2035a9352565079dc7b817f51981516ce: Status 404 returned error can't find the container with id bf959b3d6a129769748d11dc9ca956f2035a9352565079dc7b817f51981516ce
Oct 31 18:47:12.204548 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.204517    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:47:12.204621 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.204579    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:47:12.204621 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.204612    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:47:12.256952 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.256933    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"
Oct 31 18:47:12.256952 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.256937    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf" event={"ID":"7fc48784-e2e5-4d2f-90f0-9edbe9c9644f","Type":"ContainerDied","Data":"4df1b658c8ab12e37a7237660f263685e22dfd8441366dea0b756978b49e9e1f"}
Oct 31 18:47:12.257380 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.257007    4484 scope.go:122] "RemoveContainer" containerID="0a06e241445c11c5c98dd80787ae60f68668d565e813a1db0cb78e1b75e63394"
Oct 31 18:47:12.258584 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.258565    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"
Oct 31 18:47:12.258684 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.258575    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp" event={"ID":"d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f","Type":"ContainerDied","Data":"125eedbffcc16449951b9ca325f160ad78b0db29770b062ef1f64bdc91df74ab"}
Oct 31 18:47:12.259812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.259791    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" event={"ID":"0159e4a4-ad64-49f0-9256-9922e333235e","Type":"ContainerStarted","Data":"bf959b3d6a129769748d11dc9ca956f2035a9352565079dc7b817f51981516ce"}
Oct 31 18:47:12.277363 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.277257    4484 scope.go:122] "RemoveContainer" containerID="231b15f359fd930d52831d606f71f575e31459c1619462414fad41aca854b729"
Oct 31 18:47:12.426869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.426838    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"]
Oct 31 18:47:12.476971 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.476886    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-5fzfp"]
Oct 31 18:47:12.540022 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.539986    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"]
Oct 31 18:47:12.542984 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:12.542958    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-s45jf"]
Oct 31 18:47:13.269784 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:13.269750    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" event={"ID":"0159e4a4-ad64-49f0-9256-9922e333235e","Type":"ContainerStarted","Data":"b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5"}
Oct 31 18:47:13.390402 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:13.390356    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" podStartSLOduration=2.390343213 podStartE2EDuration="2.390343213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:47:13.387798623 +0000 UTC m=+1579.534248510" watchObservedRunningTime="2026-10-31 18:47:13.390343213 +0000 UTC m=+1579.536793077"
Oct 31 18:47:13.750383 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:13.750355    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:13.755910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:13.755886    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:13.994908 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:13.994878    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc48784-e2e5-4d2f-90f0-9edbe9c9644f" path="/var/lib/kubelet/pods/7fc48784-e2e5-4d2f-90f0-9edbe9c9644f/volumes"
Oct 31 18:47:13.995391 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:13.995371    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" path="/var/lib/kubelet/pods/d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f/volumes"
Oct 31 18:47:14.189880 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:14.189796    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"
Oct 31 18:47:14.189880 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:14.189870    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"
Oct 31 18:47:14.276147 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:14.276117    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:14.276968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:14.276901    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:47:15.715263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:15.715225    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:16.177820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:16.177785    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:16.354846 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:16.354804    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:16.531175 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:16.531143    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:17.072645 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:17.072606    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:19.057431 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:19.057395    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:19.416655 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:19.416566    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:19.684900 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:19.684807    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:23.838495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:23.838427    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:24.004542 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:24.004507    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:24.363131 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:24.363096    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:25.291554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:25.291521    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"]
Oct 31 18:47:25.291990 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:25.291730    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc" podUID="76ee432c-0300-43c5-8cb5-c4606ed6e23b" containerName="istio-proxy" containerID="cri-o://7e7bd195b96f9075c74a8800e29bbff7fe639679834a95f8161b32173070dfdc" gracePeriod=30
Oct 31 18:47:25.361799 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:25.361760    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:25.472279 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:47:25.472233    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:47:25.853016 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:25.852986    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:26.762775 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:26.762742    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:26.923952 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:26.923918    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:28.242261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:28.242225    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:28.741247 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:28.741217    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:29.520952 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:29.520891    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:29.672895 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:29.672865    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:30.424160 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.424133    4484 generic.go:356] "Generic (PLEG): container finished" podID="76ee432c-0300-43c5-8cb5-c4606ed6e23b" containerID="7e7bd195b96f9075c74a8800e29bbff7fe639679834a95f8161b32173070dfdc" exitCode=0
Oct 31 18:47:30.424305 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.424220    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc" event={"ID":"76ee432c-0300-43c5-8cb5-c4606ed6e23b","Type":"ContainerDied","Data":"7e7bd195b96f9075c74a8800e29bbff7fe639679834a95f8161b32173070dfdc"}
Oct 31 18:47:30.559411 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.559349    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:47:30.647530 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.647506    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-envoy\") pod \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") "
Oct 31 18:47:30.647691 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.647580    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-credential-socket\") pod \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") "
Oct 31 18:47:30.647691 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.647642    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-data\") pod \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") "
Oct 31 18:47:30.647691 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.647661    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-podinfo\") pod \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") "
Oct 31 18:47:30.647869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.647750    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-socket\") pod \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") "
Oct 31 18:47:30.647869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.647819    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-certs\") pod \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") "
Oct 31 18:47:30.647975 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.647884    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-token\" (UniqueName: \"kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-token\") pod \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") "
Oct 31 18:47:30.647975 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.647909    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-credential-socket" pod "76ee432c-0300-43c5-8cb5-c4606ed6e23b" (UID: "76ee432c-0300-43c5-8cb5-c4606ed6e23b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:30.647975 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.647926    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-kube-api-access-8j58r\" (UniqueName: \"kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-kube-api-access-8j58r\") pod \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") "
Oct 31 18:47:30.648127 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.647996    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istiod-ca-cert\") pod \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\" (UID: \"76ee432c-0300-43c5-8cb5-c4606ed6e23b\") "
Oct 31 18:47:30.648183 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.648124    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-certs" pod "76ee432c-0300-43c5-8cb5-c4606ed6e23b" (UID: "76ee432c-0300-43c5-8cb5-c4606ed6e23b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:30.648498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.648260    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-socket" pod "76ee432c-0300-43c5-8cb5-c4606ed6e23b" (UID: "76ee432c-0300-43c5-8cb5-c4606ed6e23b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:30.648498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.648310    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-data" pod "76ee432c-0300-43c5-8cb5-c4606ed6e23b" (UID: "76ee432c-0300-43c5-8cb5-c4606ed6e23b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:30.648815 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.648648    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:30.648815 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.648670    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:30.648815 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.648648    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istiod-ca-cert" pod "76ee432c-0300-43c5-8cb5-c4606ed6e23b" (UID: "76ee432c-0300-43c5-8cb5-c4606ed6e23b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:30.648815 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.648686    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:30.649027 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.648847    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:30.649825 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.649792    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-envoy" pod "76ee432c-0300-43c5-8cb5-c4606ed6e23b" (UID: "76ee432c-0300-43c5-8cb5-c4606ed6e23b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:30.650179 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.650164    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-token" pod "76ee432c-0300-43c5-8cb5-c4606ed6e23b" (UID: "76ee432c-0300-43c5-8cb5-c4606ed6e23b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:30.650303 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.650286    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-kube-api-access-8j58r" pod "76ee432c-0300-43c5-8cb5-c4606ed6e23b" (UID: "76ee432c-0300-43c5-8cb5-c4606ed6e23b"). InnerVolumeSpecName "kube-api-access-8j58r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:30.650338 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.650322    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-podinfo" pod "76ee432c-0300-43c5-8cb5-c4606ed6e23b" (UID: "76ee432c-0300-43c5-8cb5-c4606ed6e23b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:47:30.750515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.750485    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:30.750515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.750515    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j58r\" (UniqueName: \"kubernetes.io/projected/76ee432c-0300-43c5-8cb5-c4606ed6e23b-kube-api-access-8j58r\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:30.750706 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.750525    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:30.750706 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.750534    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:30.750706 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:30.750542    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ee432c-0300-43c5-8cb5-c4606ed6e23b-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:31.432777 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:31.432747    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"
Oct 31 18:47:31.432949 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:31.432774    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc" event={"ID":"76ee432c-0300-43c5-8cb5-c4606ed6e23b","Type":"ContainerDied","Data":"853914bb886370f40da4c141c5311fe489ddf63d263507772fc0f5633921e42e"}
Oct 31 18:47:31.432949 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:31.432868    4484 scope.go:122] "RemoveContainer" containerID="7e7bd195b96f9075c74a8800e29bbff7fe639679834a95f8161b32173070dfdc"
Oct 31 18:47:31.621744 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:31.621712    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"]
Oct 31 18:47:31.624690 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:31.624670    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-pvxqc"]
Oct 31 18:47:31.994614 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:31.994584    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ee432c-0300-43c5-8cb5-c4606ed6e23b" path="/var/lib/kubelet/pods/76ee432c-0300-43c5-8cb5-c4606ed6e23b/volumes"
Oct 31 18:47:32.319974 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:32.319898    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:32.903050 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:32.903014    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:33.192525 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:33.192380    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:34.543737 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:34.543691    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:34.698712 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:34.698665    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:34.862653 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:34.862569    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:35.834738 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:35.834705    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:36.161725 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:36.161642    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"]
Oct 31 18:47:36.272399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:36.272368    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"]
Oct 31 18:47:36.537014 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:36.536974    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:36.685947 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:36.685918    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:37.463829 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:37.463774    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:37.638160 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:37.638118    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:37.796357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:37.796309    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:39.111450 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:39.111423    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"]
Oct 31 18:47:39.111867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:39.111645    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" podUID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" containerName="istio-proxy" containerID="cri-o://443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518" gracePeriod=30
Oct 31 18:47:39.999198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:39.999166    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:41.162181 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.162134    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l" podUID="a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6" containerName="payload-processing" containerID="cri-o://e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e" gracePeriod=25
Oct 31 18:47:41.273639 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.273579    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v" podUID="67d2af34-9c5e-4e1b-8d1d-404259ea9ef9" containerName="payload-pre-processing" containerID="cri-o://612703de762cdc23f0157954e101ec5346697f9fd7f7e993d6c741afe43bf31a" gracePeriod=25
Oct 31 18:47:41.405636 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.405605    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:41.433750 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.433683    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"
Oct 31 18:47:41.499159 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.499130    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-kube-api-access-l5mwr\" (UniqueName: \"kubernetes.io/projected/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-kube-api-access-l5mwr\") pod \"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6\" (UID: \"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6\") "
Oct 31 18:47:41.499340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.499266    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-plugins-config-volume\") pod \"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6\" (UID: \"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6\") "
Oct 31 18:47:41.500152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.500120    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-plugins-config-volume" pod "a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6" (UID: "a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:41.502006 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.501974    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-kube-api-access-l5mwr" pod "a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6" (UID: "a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6"). InnerVolumeSpecName "kube-api-access-l5mwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:41.504729 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.504705    4484 generic.go:356] "Generic (PLEG): container finished" podID="67d2af34-9c5e-4e1b-8d1d-404259ea9ef9" containerID="612703de762cdc23f0157954e101ec5346697f9fd7f7e993d6c741afe43bf31a" exitCode=0
Oct 31 18:47:41.504845 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.504799    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v" event={"ID":"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9","Type":"ContainerDied","Data":"612703de762cdc23f0157954e101ec5346697f9fd7f7e993d6c741afe43bf31a"}
Oct 31 18:47:41.506385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.506363    4484 generic.go:356] "Generic (PLEG): container finished" podID="a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6" containerID="e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e" exitCode=0
Oct 31 18:47:41.506507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.506407    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"
Oct 31 18:47:41.506507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.506443    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l" event={"ID":"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6","Type":"ContainerDied","Data":"e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e"}
Oct 31 18:47:41.506617 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.506523    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l" event={"ID":"a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6","Type":"ContainerDied","Data":"e0238eaeda94220c905aa965664cbac80317bd25ab047e433af79746729519e6"}
Oct 31 18:47:41.506617 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.506553    4484 scope.go:122] "RemoveContainer" containerID="e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e"
Oct 31 18:47:41.528417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.528391    4484 scope.go:122] "RemoveContainer" containerID="e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e"
Oct 31 18:47:41.528797 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:47:41.528774    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e\": container with ID starting with e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e not found: ID does not exist" containerID="e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e"
Oct 31 18:47:41.528848 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.528806    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e"} err="failed to get container status \"e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e\": rpc error: code = NotFound desc = could not find container \"e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e\": container with ID starting with e449f52dabea77357663623ebc6c9fb39169f918be622d0600efca40f366a60e not found: ID does not exist"
Oct 31 18:47:41.561374 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.561351    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"
Oct 31 18:47:41.601979 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.601950    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5mwr\" (UniqueName: \"kubernetes.io/projected/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-kube-api-access-l5mwr\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:41.602106 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.601982    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:41.663805 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.663779    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"]
Oct 31 18:47:41.667844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.667540    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cd7d17-6bd9b55db5-9gs8l"]
Oct 31 18:47:41.702835 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.702780    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-kube-api-access-tph87\" (UniqueName: \"kubernetes.io/projected/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-kube-api-access-tph87\") pod \"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9\" (UID: \"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9\") "
Oct 31 18:47:41.702955 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.702846    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-plugins-config-volume\") pod \"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9\" (UID: \"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9\") "
Oct 31 18:47:41.703145 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.703123    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-plugins-config-volume" pod "67d2af34-9c5e-4e1b-8d1d-404259ea9ef9" (UID: "67d2af34-9c5e-4e1b-8d1d-404259ea9ef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:41.703210 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.703194    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:41.704875 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.704855    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-kube-api-access-tph87" pod "67d2af34-9c5e-4e1b-8d1d-404259ea9ef9" (UID: "67d2af34-9c5e-4e1b-8d1d-404259ea9ef9"). InnerVolumeSpecName "kube-api-access-tph87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:41.805231 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.805191    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tph87\" (UniqueName: \"kubernetes.io/projected/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9-kube-api-access-tph87\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:41.993795 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:41.993766    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6" path="/var/lib/kubelet/pods/a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6/volumes"
Oct 31 18:47:42.216031 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.215998    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:42.412243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.412155    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:42.518189 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.518147    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v" event={"ID":"67d2af34-9c5e-4e1b-8d1d-404259ea9ef9","Type":"ContainerDied","Data":"e0985829ba4ae763be4035eb5ff5f0422a7734465b66da342dac0d9f53901427"}
Oct 31 18:47:42.518189 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.518168    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"
Oct 31 18:47:42.518457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.518242    4484 scope.go:122] "RemoveContainer" containerID="612703de762cdc23f0157954e101ec5346697f9fd7f7e993d6c741afe43bf31a"
Oct 31 18:47:42.608342 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.608311    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"]
Oct 31 18:47:42.609030 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.609009    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6" containerName="payload-processing"
Oct 31 18:47:42.609152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.609031    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6" containerName="payload-processing"
Oct 31 18:47:42.609152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.609051    4484 cpu_manager.go:414] "containerMap: removing container" podUID="76ee432c-0300-43c5-8cb5-c4606ed6e23b" containerName="istio-proxy"
Oct 31 18:47:42.609152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.609058    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ee432c-0300-43c5-8cb5-c4606ed6e23b" containerName="istio-proxy"
Oct 31 18:47:42.609152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.609076    4484 cpu_manager.go:414] "containerMap: removing container" podUID="d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" containerName="istio-proxy"
Oct 31 18:47:42.609152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.609086    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" containerName="istio-proxy"
Oct 31 18:47:42.609152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.609099    4484 cpu_manager.go:414] "containerMap: removing container" podUID="67d2af34-9c5e-4e1b-8d1d-404259ea9ef9" containerName="payload-pre-processing"
Oct 31 18:47:42.609152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.609107    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d2af34-9c5e-4e1b-8d1d-404259ea9ef9" containerName="payload-pre-processing"
Oct 31 18:47:42.609528 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.609293    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a71ab3c6-adf2-41c5-a4b1-7e7aad8256c6" containerName="payload-processing"
Oct 31 18:47:42.609528 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.609314    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ee432c-0300-43c5-8cb5-c4606ed6e23b" containerName="istio-proxy"
Oct 31 18:47:42.609528 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.609327    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7cf4ebc-cd36-4baa-83df-2b2a9e485b9f" containerName="istio-proxy"
Oct 31 18:47:42.609528 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.609338    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d2af34-9c5e-4e1b-8d1d-404259ea9ef9" containerName="payload-pre-processing"
Oct 31 18:47:42.614190 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.614170    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"
Oct 31 18:47:42.666965 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.666894    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5d1b3f\""
Oct 31 18:47:42.667115 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.666992    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5d1b3f-dockercfg-vczl8\""
Oct 31 18:47:42.667503 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.667459    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"]
Oct 31 18:47:42.687733 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.687707    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"]
Oct 31 18:47:42.692859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.692828    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cd7d17-759547b96c-scb9v"]
Oct 31 18:47:42.718018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.717395    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpj66\" (UniqueName: \"kubernetes.io/projected/4120c0da-3927-463c-a732-40aad4707232-kube-api-access-bpj66\") pod \"payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr\" (UID: \"4120c0da-3927-463c-a732-40aad4707232\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"
Oct 31 18:47:42.718018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.717449    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4120c0da-3927-463c-a732-40aad4707232-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr\" (UID: \"4120c0da-3927-463c-a732-40aad4707232\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"
Oct 31 18:47:42.766503 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.765894    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"]
Oct 31 18:47:42.789505 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.788776    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"
Oct 31 18:47:42.819201 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.819166    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpj66\" (UniqueName: \"kubernetes.io/projected/4120c0da-3927-463c-a732-40aad4707232-kube-api-access-bpj66\") pod \"payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr\" (UID: \"4120c0da-3927-463c-a732-40aad4707232\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"
Oct 31 18:47:42.819201 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.819203    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4120c0da-3927-463c-a732-40aad4707232-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr\" (UID: \"4120c0da-3927-463c-a732-40aad4707232\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"
Oct 31 18:47:42.820156 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.820131    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4120c0da-3927-463c-a732-40aad4707232-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr\" (UID: \"4120c0da-3927-463c-a732-40aad4707232\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"
Oct 31 18:47:42.823821 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.823798    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"]
Oct 31 18:47:42.876385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.876340    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpj66\" (UniqueName: \"kubernetes.io/projected/4120c0da-3927-463c-a732-40aad4707232-kube-api-access-bpj66\") pod \"payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr\" (UID: \"4120c0da-3927-463c-a732-40aad4707232\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"
Oct 31 18:47:42.920695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.920620    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74\" (UID: \"7b89d508-6c46-4f54-a672-7f3cf4a6d96c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"
Oct 31 18:47:42.920847 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.920830    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcg7d\" (UniqueName: \"kubernetes.io/projected/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-kube-api-access-zcg7d\") pod \"payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74\" (UID: \"7b89d508-6c46-4f54-a672-7f3cf4a6d96c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"
Oct 31 18:47:42.928422 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:42.928396    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"
Oct 31 18:47:43.022545 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.022135    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcg7d\" (UniqueName: \"kubernetes.io/projected/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-kube-api-access-zcg7d\") pod \"payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74\" (UID: \"7b89d508-6c46-4f54-a672-7f3cf4a6d96c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"
Oct 31 18:47:43.022545 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.022217    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74\" (UID: \"7b89d508-6c46-4f54-a672-7f3cf4a6d96c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"
Oct 31 18:47:43.023040 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.023015    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74\" (UID: \"7b89d508-6c46-4f54-a672-7f3cf4a6d96c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"
Oct 31 18:47:43.079581 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.079549    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcg7d\" (UniqueName: \"kubernetes.io/projected/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-kube-api-access-zcg7d\") pod \"payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74\" (UID: \"7b89d508-6c46-4f54-a672-7f3cf4a6d96c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"
Oct 31 18:47:43.112516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.112464    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"
Oct 31 18:47:43.309427 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.309401    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"]
Oct 31 18:47:43.310193 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:47:43.310172    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4120c0da_3927_463c_a732_40aad4707232.slice/crio-47bdc4c2c6403e70dee3bcbf96ebd153de1df570e07074a76ff5b8ce10cf2335 WatchSource:0}: Error finding container 47bdc4c2c6403e70dee3bcbf96ebd153de1df570e07074a76ff5b8ce10cf2335: Status 404 returned error can't find the container with id 47bdc4c2c6403e70dee3bcbf96ebd153de1df570e07074a76ff5b8ce10cf2335
Oct 31 18:47:43.487349 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.487322    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"]
Oct 31 18:47:43.488074 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:47:43.488046    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b89d508_6c46_4f54_a672_7f3cf4a6d96c.slice/crio-281ac219f83c803dd463517b17e1db2c71ae186e9868b4aaa69a8dcabee0e71c WatchSource:0}: Error finding container 281ac219f83c803dd463517b17e1db2c71ae186e9868b4aaa69a8dcabee0e71c: Status 404 returned error can't find the container with id 281ac219f83c803dd463517b17e1db2c71ae186e9868b4aaa69a8dcabee0e71c
Oct 31 18:47:43.527282 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.527256    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74" event={"ID":"7b89d508-6c46-4f54-a672-7f3cf4a6d96c","Type":"ContainerStarted","Data":"281ac219f83c803dd463517b17e1db2c71ae186e9868b4aaa69a8dcabee0e71c"}
Oct 31 18:47:43.529689 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.529665    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr" event={"ID":"4120c0da-3927-463c-a732-40aad4707232","Type":"ContainerStarted","Data":"d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9"}
Oct 31 18:47:43.529801 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.529706    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr" event={"ID":"4120c0da-3927-463c-a732-40aad4707232","Type":"ContainerStarted","Data":"47bdc4c2c6403e70dee3bcbf96ebd153de1df570e07074a76ff5b8ce10cf2335"}
Oct 31 18:47:43.529801 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.529732    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"
Oct 31 18:47:43.645032 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.644973    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr" podStartSLOduration=1.64495888 podStartE2EDuration="1.64495888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:47:43.643810374 +0000 UTC m=+1609.790260238" watchObservedRunningTime="2026-10-31 18:47:43.64495888 +0000 UTC m=+1609.791408743"
Oct 31 18:47:43.994300 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:43.994272    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d2af34-9c5e-4e1b-8d1d-404259ea9ef9" path="/var/lib/kubelet/pods/67d2af34-9c5e-4e1b-8d1d-404259ea9ef9/volumes"
Oct 31 18:47:44.088176 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.088151    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:47:44.088330 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.088195    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" podUID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:47:44.385644 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.385617    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:47:44.438660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.438633    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istiod-ca-cert\") pod \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") "
Oct 31 18:47:44.438802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.438692    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-certs\") pod \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") "
Oct 31 18:47:44.438802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.438727    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-kube-api-access-czxqv\" (UniqueName: \"kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-kube-api-access-czxqv\") pod \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") "
Oct 31 18:47:44.438802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.438771    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-envoy\") pod \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") "
Oct 31 18:47:44.438942 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.438802    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-token\" (UniqueName: \"kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-token\") pod \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") "
Oct 31 18:47:44.438942 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.438863    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-podinfo\") pod \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") "
Oct 31 18:47:44.438942 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.438932    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-credential-socket\") pod \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") "
Oct 31 18:47:44.439090 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.439004    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-certs" pod "8fd5dbc4-d25a-4018-866e-2161b2378eaa" (UID: "8fd5dbc4-d25a-4018-866e-2161b2378eaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:44.439090 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.439035    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istiod-ca-cert" pod "8fd5dbc4-d25a-4018-866e-2161b2378eaa" (UID: "8fd5dbc4-d25a-4018-866e-2161b2378eaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:47:44.439194 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.439100    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-data\") pod \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") "
Oct 31 18:47:44.439194 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.439127    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-socket\") pod \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\" (UID: \"8fd5dbc4-d25a-4018-866e-2161b2378eaa\") "
Oct 31 18:47:44.439428 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.439259    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-credential-socket" pod "8fd5dbc4-d25a-4018-866e-2161b2378eaa" (UID: "8fd5dbc4-d25a-4018-866e-2161b2378eaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:44.439506 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.439426    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-data" pod "8fd5dbc4-d25a-4018-866e-2161b2378eaa" (UID: "8fd5dbc4-d25a-4018-866e-2161b2378eaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:44.439607 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.439563    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-socket" pod "8fd5dbc4-d25a-4018-866e-2161b2378eaa" (UID: "8fd5dbc4-d25a-4018-866e-2161b2378eaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:44.440007 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.439986    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:44.440107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.440013    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:44.440107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.440030    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:44.440107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.440046    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:44.440107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.440060    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:44.441175 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.441153    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-envoy" pod "8fd5dbc4-d25a-4018-866e-2161b2378eaa" (UID: "8fd5dbc4-d25a-4018-866e-2161b2378eaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:47:44.441274 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.441255    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-token" pod "8fd5dbc4-d25a-4018-866e-2161b2378eaa" (UID: "8fd5dbc4-d25a-4018-866e-2161b2378eaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:44.441274 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.441254    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-podinfo" pod "8fd5dbc4-d25a-4018-866e-2161b2378eaa" (UID: "8fd5dbc4-d25a-4018-866e-2161b2378eaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:47:44.441368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.441288    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-kube-api-access-czxqv" pod "8fd5dbc4-d25a-4018-866e-2161b2378eaa" (UID: "8fd5dbc4-d25a-4018-866e-2161b2378eaa"). InnerVolumeSpecName "kube-api-access-czxqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:47:44.539134 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.539041    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74" event={"ID":"7b89d508-6c46-4f54-a672-7f3cf4a6d96c","Type":"ContainerStarted","Data":"8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072"}
Oct 31 18:47:44.539134 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.539109    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"
Oct 31 18:47:44.540785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.540755    4484 generic.go:356] "Generic (PLEG): container finished" podID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" containerID="443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518" exitCode=0
Oct 31 18:47:44.541065 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.541034    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"
Oct 31 18:47:44.541185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.541077    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czxqv\" (UniqueName: \"kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-kube-api-access-czxqv\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:44.541185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.541101    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:44.541185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.541113    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:44.541185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.541124    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fd5dbc4-d25a-4018-866e-2161b2378eaa-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:47:44.541185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.541149    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" event={"ID":"8fd5dbc4-d25a-4018-866e-2161b2378eaa","Type":"ContainerDied","Data":"443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518"}
Oct 31 18:47:44.541431 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.541191    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr" event={"ID":"8fd5dbc4-d25a-4018-866e-2161b2378eaa","Type":"ContainerDied","Data":"895f6ff07e8e84390e4845c6cb045746869a8ed02430d7d97f886ae8ac91217a"}
Oct 31 18:47:44.541431 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.541219    4484 scope.go:122] "RemoveContainer" containerID="443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518"
Oct 31 18:47:44.555666 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.555651    4484 scope.go:122] "RemoveContainer" containerID="443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518"
Oct 31 18:47:44.555941 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:47:44.555924    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518\": container with ID starting with 443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518 not found: ID does not exist" containerID="443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518"
Oct 31 18:47:44.555989 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.555949    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518"} err="failed to get container status \"443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518\": rpc error: code = NotFound desc = could not find container \"443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518\": container with ID starting with 443649b4a5871de5c390ff6376f0b44e7971b2630fb13b692bc08c3f01ac8518 not found: ID does not exist"
Oct 31 18:47:44.656333 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.656284    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74" podStartSLOduration=2.656271175 podStartE2EDuration="2.656271175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:47:44.653030604 +0000 UTC m=+1610.799480472" watchObservedRunningTime="2026-10-31 18:47:44.656271175 +0000 UTC m=+1610.802721055"
Oct 31 18:47:44.765983 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.765947    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"]
Oct 31 18:47:44.771438 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:44.771413    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-pr8gr"]
Oct 31 18:47:45.248980 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:45.248947    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:45.995812 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:45.995782    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" path="/var/lib/kubelet/pods/8fd5dbc4-d25a-4018-866e-2161b2378eaa/volumes"
Oct 31 18:47:49.166803 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:49.166769    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:50.386003 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:50.385974    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:50.403592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:50.403557    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:47:53.793904 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:53.793870    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:54.478955 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:54.478925    4484 scope.go:122] "RemoveContainer" containerID="167753de7fc809b5e46612c7e051a72bee8febac30fabddd620d61a9f8a80d76"
Oct 31 18:47:54.545391 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:54.545367    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"
Oct 31 18:47:55.154001 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:55.153948    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:47:55.468270 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:47:55.468188    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:47:55.551200 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:55.551173    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"
Oct 31 18:47:59.864613 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:47:59.864567    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:04.410201 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:04.410167    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:04.588130 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:04.588095    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:04.738301 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:04.738266    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:09.364835 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:09.364793    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:11.044376 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:11.044351    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"]
Oct 31 18:48:11.094927 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:11.094901    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64e37a8f-7lwmw"]
Oct 31 18:48:11.995136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:11.995106    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d879555f-653a-47b5-8d70-bf28df4242b7" path="/var/lib/kubelet/pods/d879555f-653a-47b5-8d70-bf28df4242b7/volumes"
Oct 31 18:48:13.357402 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.357360    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"]
Oct 31 18:48:13.357931 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.357687    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds" podUID="3c09c680-68f0-46b1-becd-01129885f5e7" containerName="maas-api" containerID="cri-o://bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9" gracePeriod=30
Oct 31 18:48:13.476493 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.476445    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:13.643280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.643258    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:48:13.708290 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.708263    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-metrics-certs\") pod \"3c09c680-68f0-46b1-becd-01129885f5e7\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") "
Oct 31 18:48:13.708434 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.708405    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c09c680-68f0-46b1-becd-01129885f5e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c09c680-68f0-46b1-becd-01129885f5e7-openshift-service-ca\") pod \"3c09c680-68f0-46b1-becd-01129885f5e7\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") "
Oct 31 18:48:13.708434 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.708428    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c09c680-68f0-46b1-becd-01129885f5e7-kube-api-access-nzk9z\" (UniqueName: \"kubernetes.io/projected/3c09c680-68f0-46b1-becd-01129885f5e7-kube-api-access-nzk9z\") pod \"3c09c680-68f0-46b1-becd-01129885f5e7\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") "
Oct 31 18:48:13.708569 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.708554    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-maas-api-tls\") pod \"3c09c680-68f0-46b1-becd-01129885f5e7\" (UID: \"3c09c680-68f0-46b1-becd-01129885f5e7\") "
Oct 31 18:48:13.708817 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.708794    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c09c680-68f0-46b1-becd-01129885f5e7-openshift-service-ca" pod "3c09c680-68f0-46b1-becd-01129885f5e7" (UID: "3c09c680-68f0-46b1-becd-01129885f5e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:48:13.709095 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.709076    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c09c680-68f0-46b1-becd-01129885f5e7-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:13.710777 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.710752    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-maas-api-tls" pod "3c09c680-68f0-46b1-becd-01129885f5e7" (UID: "3c09c680-68f0-46b1-becd-01129885f5e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:48:13.710874 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.710799    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-metrics-certs" pod "3c09c680-68f0-46b1-becd-01129885f5e7" (UID: "3c09c680-68f0-46b1-becd-01129885f5e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:48:13.710874 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.710805    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c09c680-68f0-46b1-becd-01129885f5e7-kube-api-access-nzk9z" pod "3c09c680-68f0-46b1-becd-01129885f5e7" (UID: "3c09c680-68f0-46b1-becd-01129885f5e7"). InnerVolumeSpecName "kube-api-access-nzk9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:13.739925 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.739861    4484 generic.go:356] "Generic (PLEG): container finished" podID="3c09c680-68f0-46b1-becd-01129885f5e7" containerID="bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9" exitCode=0
Oct 31 18:48:13.740118 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.740035    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds" event={"ID":"3c09c680-68f0-46b1-becd-01129885f5e7","Type":"ContainerDied","Data":"bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9"}
Oct 31 18:48:13.740185 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.740131    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds" event={"ID":"3c09c680-68f0-46b1-becd-01129885f5e7","Type":"ContainerDied","Data":"5cfcb1070a1362d086df52f447b2d04f432615fd468217fbc79df5bf83e64cde"}
Oct 31 18:48:13.740240 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.740183    4484 scope.go:122] "RemoveContainer" containerID="bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9"
Oct 31 18:48:13.740380 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.740352    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"
Oct 31 18:48:13.777837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.777809    4484 scope.go:122] "RemoveContainer" containerID="bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9"
Oct 31 18:48:13.779132 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:48:13.778314    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9\": container with ID starting with bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9 not found: ID does not exist" containerID="bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9"
Oct 31 18:48:13.779132 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.778359    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9"} err="failed to get container status \"bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9\": rpc error: code = NotFound desc = could not find container \"bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9\": container with ID starting with bb7d278224ca2f38ed8cda9d2d54a28ac7c82cb1eee73bfab5b2c40b7e1615f9 not found: ID does not exist"
Oct 31 18:48:13.810626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.810594    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:13.810626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.810622    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c680-68f0-46b1-becd-01129885f5e7-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:13.810827 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.810636    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzk9z\" (UniqueName: \"kubernetes.io/projected/3c09c680-68f0-46b1-becd-01129885f5e7-kube-api-access-nzk9z\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:13.932307 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.931976    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"]
Oct 31 18:48:13.935587 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.935558    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cd7d17-65b8bdbd9f-bw8ds"]
Oct 31 18:48:13.994574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:13.994543    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c09c680-68f0-46b1-becd-01129885f5e7" path="/var/lib/kubelet/pods/3c09c680-68f0-46b1-becd-01129885f5e7/volumes"
Oct 31 18:48:15.249994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:15.249963    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:16.308594 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:16.308564    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"]
Oct 31 18:48:16.309010 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:16.308731    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerName="istio-proxy" containerID="cri-o://4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7" gracePeriod=30
Oct 31 18:48:16.358468 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:16.358439    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"]
Oct 31 18:48:16.358684 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:16.358661    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" podUID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" containerName="istio-proxy" containerID="cri-o://9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2" gracePeriod=30
Oct 31 18:48:17.503394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.503353    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:17.644687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.644657    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"]
Oct 31 18:48:17.645254 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.645241    4484 cpu_manager.go:414] "containerMap: removing container" podUID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" containerName="istio-proxy"
Oct 31 18:48:17.645297 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.645255    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" containerName="istio-proxy"
Oct 31 18:48:17.645297 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.645271    4484 cpu_manager.go:414] "containerMap: removing container" podUID="3c09c680-68f0-46b1-becd-01129885f5e7" containerName="maas-api"
Oct 31 18:48:17.645297 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.645276    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c09c680-68f0-46b1-becd-01129885f5e7" containerName="maas-api"
Oct 31 18:48:17.645398 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.645385    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd5dbc4-d25a-4018-866e-2161b2378eaa" containerName="istio-proxy"
Oct 31 18:48:17.645398 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.645395    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c09c680-68f0-46b1-becd-01129885f5e7" containerName="maas-api"
Oct 31 18:48:17.649051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.649033    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.701572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.701551    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3417b822-openshift-default-dockercfg-hx6x8\""
Oct 31 18:48:17.755514 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.755431    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"]
Oct 31 18:48:17.781514 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.781467    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.781648 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.781547    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.781687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.781661    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.781746 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.781733    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.781785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.781754    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gc7\" (UniqueName: \"kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-kube-api-access-22gc7\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.781785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.781777    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.781859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.781793    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.781893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.781887    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.781939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.781929    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.883224 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.883191    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.883224 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.883222    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.883461 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.883256    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.883461 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.883296    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.883461 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.883370    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.883461 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.883419    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.883734 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.883548    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.883734 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.883610    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.883734 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.883635    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gc7\" (UniqueName: \"kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-kube-api-access-22gc7\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.883734 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.883670    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.883734 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.883692    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.883982 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.883933    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.884170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.884151    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.884384 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.884366    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.886618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.886579    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.886891 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.886869    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.940781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.940756    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.940939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.940920    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gc7\" (UniqueName: \"kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-kube-api-access-22gc7\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:17.962722 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:17.962703    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:18.387580 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:18.387502    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"]
Oct 31 18:48:18.425946 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:48:18.425914    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab42d63_0089_4fdd_9d07_441ba7c11b80.slice/crio-2df4c0160f11e6822b190d86b10350411bef0010a53c6a79f49b958ce6872cfa WatchSource:0}: Error finding container 2df4c0160f11e6822b190d86b10350411bef0010a53c6a79f49b958ce6872cfa: Status 404 returned error can't find the container with id 2df4c0160f11e6822b190d86b10350411bef0010a53c6a79f49b958ce6872cfa
Oct 31 18:48:18.428309 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:18.428276    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:48:18.428382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:18.428345    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:48:18.428424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:18.428378    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:48:18.780935 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:18.780905    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh" event={"ID":"5ab42d63-0089-4fdd-9d07-441ba7c11b80","Type":"ContainerStarted","Data":"25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49"}
Oct 31 18:48:18.781357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:18.780954    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh" event={"ID":"5ab42d63-0089-4fdd-9d07-441ba7c11b80","Type":"ContainerStarted","Data":"2df4c0160f11e6822b190d86b10350411bef0010a53c6a79f49b958ce6872cfa"}
Oct 31 18:48:18.900921 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:18.900867    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh" podStartSLOduration=1.900852913 podStartE2EDuration="1.900852913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:18.898176415 +0000 UTC m=+1645.044626280" watchObservedRunningTime="2026-10-31 18:48:18.900852913 +0000 UTC m=+1645.047302776"
Oct 31 18:48:18.973673 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:18.973580    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:19.985704 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:19.985666    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 18:48:19.986180 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:19.985735    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh" podUID="5ab42d63-0089-4fdd-9d07-441ba7c11b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 18:48:19.992090 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:19.992056    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:20.188499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:20.188276    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:48:20.188499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:20.188333    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:48:20.972338 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:20.971876    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 18:48:20.972338 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:20.971936    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh" podUID="5ab42d63-0089-4fdd-9d07-441ba7c11b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 18:48:21.702687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.702524    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:48:21.727125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.727103    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:48:21.727499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.727465    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-credential-socket\") pod \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") "
Oct 31 18:48:21.727546 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.727517    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-kube-api-access-cq7ng\" (UniqueName: \"kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-kube-api-access-cq7ng\") pod \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") "
Oct 31 18:48:21.727598 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.727543    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-socket\") pod \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") "
Oct 31 18:48:21.727598 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.727563    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-podinfo\") pod \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") "
Oct 31 18:48:21.727660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.727601    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istiod-ca-cert\") pod \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") "
Oct 31 18:48:21.727660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.727629    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-certs\") pod \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") "
Oct 31 18:48:21.727720 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.727671    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-token\" (UniqueName: \"kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-token\") pod \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") "
Oct 31 18:48:21.727775 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.727764    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-envoy\") pod \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") "
Oct 31 18:48:21.727806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.727786    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-data\") pod \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\" (UID: \"217d6077-722f-4e9a-b9ee-d3e68ce59fa1\") "
Oct 31 18:48:21.728504 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.728459    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-data" pod "217d6077-722f-4e9a-b9ee-d3e68ce59fa1" (UID: "217d6077-722f-4e9a-b9ee-d3e68ce59fa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:21.728627 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.728599    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-credential-socket" pod "217d6077-722f-4e9a-b9ee-d3e68ce59fa1" (UID: "217d6077-722f-4e9a-b9ee-d3e68ce59fa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:21.728704 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.728607    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-socket" pod "217d6077-722f-4e9a-b9ee-d3e68ce59fa1" (UID: "217d6077-722f-4e9a-b9ee-d3e68ce59fa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:21.728819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.728794    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istiod-ca-cert" pod "217d6077-722f-4e9a-b9ee-d3e68ce59fa1" (UID: "217d6077-722f-4e9a-b9ee-d3e68ce59fa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:48:21.729018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.728992    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-certs" pod "217d6077-722f-4e9a-b9ee-d3e68ce59fa1" (UID: "217d6077-722f-4e9a-b9ee-d3e68ce59fa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:21.730174 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.730145    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-podinfo" pod "217d6077-722f-4e9a-b9ee-d3e68ce59fa1" (UID: "217d6077-722f-4e9a-b9ee-d3e68ce59fa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:48:21.730453 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.730430    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-envoy" pod "217d6077-722f-4e9a-b9ee-d3e68ce59fa1" (UID: "217d6077-722f-4e9a-b9ee-d3e68ce59fa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:21.730453 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.730439    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-kube-api-access-cq7ng" pod "217d6077-722f-4e9a-b9ee-d3e68ce59fa1" (UID: "217d6077-722f-4e9a-b9ee-d3e68ce59fa1"). InnerVolumeSpecName "kube-api-access-cq7ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:21.730646 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.730630    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-token" pod "217d6077-722f-4e9a-b9ee-d3e68ce59fa1" (UID: "217d6077-722f-4e9a-b9ee-d3e68ce59fa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:21.792207 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.792166    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:21.803641 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.803612    4484 generic.go:356] "Generic (PLEG): container finished" podID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" containerID="9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2" exitCode=0
Oct 31 18:48:21.803782 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.803665    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"
Oct 31 18:48:21.803782 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.803693    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" event={"ID":"3e4b3792-3dcd-4def-972e-a2876ecc56cd","Type":"ContainerDied","Data":"9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2"}
Oct 31 18:48:21.803891 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.803792    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b" event={"ID":"3e4b3792-3dcd-4def-972e-a2876ecc56cd","Type":"ContainerDied","Data":"719afe8c7f60081e7691b7067118ff366f0f616d6e6fe515a35b3877321cd3e4"}
Oct 31 18:48:21.803891 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.803826    4484 scope.go:122] "RemoveContainer" containerID="9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2"
Oct 31 18:48:21.805299 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.805280    4484 generic.go:356] "Generic (PLEG): container finished" podID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerID="4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7" exitCode=0
Oct 31 18:48:21.805434 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.805328    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"
Oct 31 18:48:21.805434 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.805372    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" event={"ID":"217d6077-722f-4e9a-b9ee-d3e68ce59fa1","Type":"ContainerDied","Data":"4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7"}
Oct 31 18:48:21.805434 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.805414    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k" event={"ID":"217d6077-722f-4e9a-b9ee-d3e68ce59fa1","Type":"ContainerDied","Data":"e4ada3600a8360dad24dbe551ad06d48459063c14c1df2dd41872c5e4fb52cc9"}
Oct 31 18:48:21.814025 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.814011    4484 scope.go:122] "RemoveContainer" containerID="9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2"
Oct 31 18:48:21.814299 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:48:21.814279    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2\": container with ID starting with 9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2 not found: ID does not exist" containerID="9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2"
Oct 31 18:48:21.814342 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.814307    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2"} err="failed to get container status \"9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2\": rpc error: code = NotFound desc = could not find container \"9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2\": container with ID starting with 9e9ce469801cf1e333d1a6e0b8565db6d73e8b4854ba8e678a2fc00a2595c1d2 not found: ID does not exist"
Oct 31 18:48:21.814342 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.814323    4484 scope.go:122] "RemoveContainer" containerID="4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7"
Oct 31 18:48:21.823383 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.823362    4484 scope.go:122] "RemoveContainer" containerID="4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7"
Oct 31 18:48:21.823644 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:48:21.823626    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7\": container with ID starting with 4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7 not found: ID does not exist" containerID="4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7"
Oct 31 18:48:21.823694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.823650    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7"} err="failed to get container status \"4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7\": rpc error: code = NotFound desc = could not find container \"4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7\": container with ID starting with 4546cc60c463b81a341c57f77fa4511321f0d2021763949105ee6f754c9414f7 not found: ID does not exist"
Oct 31 18:48:21.829140 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.829116    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-token\" (UniqueName: \"kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-token\") pod \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") "
Oct 31 18:48:21.829257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.829189    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-credential-socket\") pod \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") "
Oct 31 18:48:21.829257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.829248    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-envoy\") pod \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") "
Oct 31 18:48:21.829397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.829379    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istiod-ca-cert\") pod \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") "
Oct 31 18:48:21.829456 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.829416    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-podinfo\") pod \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") "
Oct 31 18:48:21.829575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.829557    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-credential-socket" pod "3e4b3792-3dcd-4def-972e-a2876ecc56cd" (UID: "3e4b3792-3dcd-4def-972e-a2876ecc56cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:21.829631 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.829609    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-certs\") pod \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") "
Oct 31 18:48:21.829706 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.829689    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-kube-api-access-8bqwz\" (UniqueName: \"kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-kube-api-access-8bqwz\") pod \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") "
Oct 31 18:48:21.829799 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.829784    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-socket\") pod \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") "
Oct 31 18:48:21.829849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.829813    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istiod-ca-cert" pod "3e4b3792-3dcd-4def-972e-a2876ecc56cd" (UID: "3e4b3792-3dcd-4def-972e-a2876ecc56cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.829870    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-data\") pod \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\" (UID: \"3e4b3792-3dcd-4def-972e-a2876ecc56cd\") "
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830387    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830409    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830424    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq7ng\" (UniqueName: \"kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-kube-api-access-cq7ng\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830439    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830441    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-certs" pod "3e4b3792-3dcd-4def-972e-a2876ecc56cd" (UID: "3e4b3792-3dcd-4def-972e-a2876ecc56cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830499    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830516    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830531    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830544    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830558    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830570    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/217d6077-722f-4e9a-b9ee-d3e68ce59fa1-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830583    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830793    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-data" pod "3e4b3792-3dcd-4def-972e-a2876ecc56cd" (UID: "3e4b3792-3dcd-4def-972e-a2876ecc56cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:21.831994 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.830974    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-socket" pod "3e4b3792-3dcd-4def-972e-a2876ecc56cd" (UID: "3e4b3792-3dcd-4def-972e-a2876ecc56cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:21.833669 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.833645    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-envoy" pod "3e4b3792-3dcd-4def-972e-a2876ecc56cd" (UID: "3e4b3792-3dcd-4def-972e-a2876ecc56cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:21.833803 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.833784    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-podinfo" pod "3e4b3792-3dcd-4def-972e-a2876ecc56cd" (UID: "3e4b3792-3dcd-4def-972e-a2876ecc56cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:48:21.833921 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.833904    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-kube-api-access-8bqwz" pod "3e4b3792-3dcd-4def-972e-a2876ecc56cd" (UID: "3e4b3792-3dcd-4def-972e-a2876ecc56cd"). InnerVolumeSpecName "kube-api-access-8bqwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:21.834091 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.834071    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-token" pod "3e4b3792-3dcd-4def-972e-a2876ecc56cd" (UID: "3e4b3792-3dcd-4def-972e-a2876ecc56cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:21.932024 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.931938    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.932024 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.931969    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.932024 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.931982    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.932024 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.931994    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.932024 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.932006    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bqwz\" (UniqueName: \"kubernetes.io/projected/3e4b3792-3dcd-4def-972e-a2876ecc56cd-kube-api-access-8bqwz\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.932024 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.932020    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.932313 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.932033    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4b3792-3dcd-4def-972e-a2876ecc56cd-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:21.985556 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.985162    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 18:48:21.985556 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.985230    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh" podUID="5ab42d63-0089-4fdd-9d07-441ba7c11b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 18:48:21.985556 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:21.985300    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:22.114637 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:22.114123    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"]
Oct 31 18:48:22.117443 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:22.117414    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-59f4k"]
Oct 31 18:48:22.232306 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:22.232214    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"]
Oct 31 18:48:22.235859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:22.235835    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-lc96b"]
Oct 31 18:48:22.981504 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:22.975136    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 18:48:22.981504 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:22.975189    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh" podUID="5ab42d63-0089-4fdd-9d07-441ba7c11b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 18:48:23.291453 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:23.291420    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:23.721736 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:23.721658    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:23.973558 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:23.966941    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 18:48:23.973558 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:23.966996    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh" podUID="5ab42d63-0089-4fdd-9d07-441ba7c11b80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 18:48:23.995029 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:23.994998    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" path="/var/lib/kubelet/pods/217d6077-722f-4e9a-b9ee-d3e68ce59fa1/volumes"
Oct 31 18:48:23.995551 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:23.995527    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" path="/var/lib/kubelet/pods/3e4b3792-3dcd-4def-972e-a2876ecc56cd/volumes"
Oct 31 18:48:24.654548 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.654505    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:24.742231 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.742198    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"]
Oct 31 18:48:24.742800 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.742787    4484 cpu_manager.go:414] "containerMap: removing container" podUID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" containerName="istio-proxy"
Oct 31 18:48:24.742839 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.742802    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" containerName="istio-proxy"
Oct 31 18:48:24.742839 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.742834    4484 cpu_manager.go:414] "containerMap: removing container" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerName="istio-proxy"
Oct 31 18:48:24.742839 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.742839    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerName="istio-proxy"
Oct 31 18:48:24.742949 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.742939    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="217d6077-722f-4e9a-b9ee-d3e68ce59fa1" containerName="istio-proxy"
Oct 31 18:48:24.742977 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.742952    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e4b3792-3dcd-4def-972e-a2876ecc56cd" containerName="istio-proxy"
Oct 31 18:48:24.752724 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.752690    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:24.792972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.792939    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-metrics-certs\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:24.793139 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.793098    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:24.793184 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.793165    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:24.793242 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.793210    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vtv\" (UniqueName: \"kubernetes.io/projected/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-kube-api-access-q5vtv\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:24.803189 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.803159    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"]
Oct 31 18:48:24.849682 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.849652    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5d1b3f-serving-cert\""
Oct 31 18:48:24.895756 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.895723    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:24.895970 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.895819    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:24.895970 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.895852    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vtv\" (UniqueName: \"kubernetes.io/projected/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-kube-api-access-q5vtv\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:24.896096 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.896002    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-metrics-certs\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:24.896600 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:48:24.896577    4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-serving-cert: secret "maas-api-e2e-ipp-a-5d1b3f-serving-cert" not found
Oct 31 18:48:24.896830 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:48:24.896814    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-maas-api-tls podName:24f9a5e2-42a9-4f4a-9289-8bb7699eb450 nodeName:}" failed. No retries permitted until 2026-10-31 18:48:25.39679102 +0000 UTC m=+1651.543240882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-maas-api-tls") pod "maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd" (UID: "24f9a5e2-42a9-4f4a-9289-8bb7699eb450") : secret "maas-api-e2e-ipp-a-5d1b3f-serving-cert" not found
Oct 31 18:48:24.896944 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.896579    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:24.899389 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.899363    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-metrics-certs\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:24.954634 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.954550    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vtv\" (UniqueName: \"kubernetes.io/projected/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-kube-api-access-q5vtv\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:24.955909 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:24.955885    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:25.107779 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:25.107745    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:25.193447 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:25.193412    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:25.194332 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:25.194287    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:25.195342 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:25.195324    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:48:25.266892 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:25.266857    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:25.430542 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:25.430506    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:25.447901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:25.447863    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:25.452069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:25.452044    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:25.504537 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:48:25.503910    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:48:25.595429 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:25.595345    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:25.667866 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:25.667832    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:26.067881 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:26.067857    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"]
Oct 31 18:48:26.069074 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:48:26.069039    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f9a5e2_42a9_4f4a_9289_8bb7699eb450.slice/crio-b4687efb21871a650cb614c2997df0b5ad4013f70a114e9cf4c2a1ed6ae6f559 WatchSource:0}: Error finding container b4687efb21871a650cb614c2997df0b5ad4013f70a114e9cf4c2a1ed6ae6f559: Status 404 returned error can't find the container with id b4687efb21871a650cb614c2997df0b5ad4013f70a114e9cf4c2a1ed6ae6f559
Oct 31 18:48:26.070438 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:26.070418    4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 18:48:26.848829 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:26.848795    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd" event={"ID":"24f9a5e2-42a9-4f4a-9289-8bb7699eb450","Type":"ContainerStarted","Data":"b4687efb21871a650cb614c2997df0b5ad4013f70a114e9cf4c2a1ed6ae6f559"}
Oct 31 18:48:27.856282 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:27.856247    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd" event={"ID":"24f9a5e2-42a9-4f4a-9289-8bb7699eb450","Type":"ContainerStarted","Data":"74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6"}
Oct 31 18:48:27.856749 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:27.856461    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:27.974688 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:27.974638    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd" podStartSLOduration=3.162486536 podStartE2EDuration="3.974623103s" totalImagesPullingTime="812.136567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:27.971521724 +0000 UTC m=+1654.117971588" watchObservedRunningTime="2026-10-31 18:48:27.974623103 +0000 UTC m=+1654.121072966"
Oct 31 18:48:30.191231 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:30.191197    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:30.602979 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:30.602945    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:30.781860 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:30.781822    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:31.043429 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:31.043401    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:32.764052 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.764021    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"]
Oct 31 18:48:32.768295 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.768276    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:32.798524 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.798503    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7557-5a8696"]
Oct 31 18:48:32.802417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.802400    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7557-5a8696"
Oct 31 18:48:32.855257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.855231    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bg2\""
Oct 31 18:48:32.901917 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.901887    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7557-5a8696"]
Oct 31 18:48:32.940796 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.940742    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:32.940796 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.940782    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:32.940987 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.940893    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:32.940987 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.940917    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:32.940987 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.940973    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d452ef-90cf-4ec0-932d-73a0ec64030e-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:32.941078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.941069    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:32.941111 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.941101    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng7rc\" (UniqueName: \"kubernetes.io/projected/a0496eb0-bb46-48dc-be20-b3345323f146-kube-api-access-ng7rc\") pod \"test-curl-7557-5a8696\" (UID: \"a0496eb0-bb46-48dc-be20-b3345323f146\") " pod="opendatahub/test-curl-7557-5a8696"
Oct 31 18:48:32.941209 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.941193    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:32.941263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.941250    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t72sk\" (UniqueName: \"kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-kube-api-access-t72sk\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:32.941344 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.941329    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:32.969736 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:32.969706    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"]
Oct 31 18:48:33.042319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042240    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.042511 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042320    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t72sk\" (UniqueName: \"kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-kube-api-access-t72sk\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.042511 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042390    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.042511 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042415    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.042511 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042441    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.042739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042541    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.042739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042579    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.042739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042628    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d452ef-90cf-4ec0-932d-73a0ec64030e-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.042739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042712    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.042739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042727    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.043050 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042752    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng7rc\" (UniqueName: \"kubernetes.io/projected/a0496eb0-bb46-48dc-be20-b3345323f146-kube-api-access-ng7rc\") pod \"test-curl-7557-5a8696\" (UID: \"a0496eb0-bb46-48dc-be20-b3345323f146\") " pod="opendatahub/test-curl-7557-5a8696"
Oct 31 18:48:33.043050 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042817    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.043050 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.042936    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.043197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.043087    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.043590 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.043573    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d452ef-90cf-4ec0-932d-73a0ec64030e-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.044862 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.044845    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.045091 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.045075    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.100174 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.100124    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng7rc\" (UniqueName: \"kubernetes.io/projected/a0496eb0-bb46-48dc-be20-b3345323f146-kube-api-access-ng7rc\") pod \"test-curl-7557-5a8696\" (UID: \"a0496eb0-bb46-48dc-be20-b3345323f146\") " pod="opendatahub/test-curl-7557-5a8696"
Oct 31 18:48:33.100174 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.100167    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.100356 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.100288    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t72sk\" (UniqueName: \"kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-kube-api-access-t72sk\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.113776 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.113757    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7557-5a8696"
Oct 31 18:48:33.381053 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.380989    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:33.486672 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:48:33.486629    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0496eb0_bb46_48dc_be20_b3345323f146.slice/crio-46a9cf36e0edcd9a13600ee6d258177dde624fc6e99d6fd5add1fb1056842027 WatchSource:0}: Error finding container 46a9cf36e0edcd9a13600ee6d258177dde624fc6e99d6fd5add1fb1056842027: Status 404 returned error can't find the container with id 46a9cf36e0edcd9a13600ee6d258177dde624fc6e99d6fd5add1fb1056842027
Oct 31 18:48:33.487017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.486996    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7557-5a8696"]
Oct 31 18:48:33.807107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.807084    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"]
Oct 31 18:48:33.807493 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:48:33.807296    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d452ef_90cf_4ec0_932d_73a0ec64030e.slice/crio-8c941ced82a2697d8814b34fe8ff773b00f345e50b881843e4c722c3061b4876 WatchSource:0}: Error finding container 8c941ced82a2697d8814b34fe8ff773b00f345e50b881843e4c722c3061b4876: Status 404 returned error can't find the container with id 8c941ced82a2697d8814b34fe8ff773b00f345e50b881843e4c722c3061b4876
Oct 31 18:48:33.809758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.809728    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:48:33.809859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.809788    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:48:33.809859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.809815    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:48:33.872005 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.871975    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:48:33.919111 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.919077    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" event={"ID":"96d452ef-90cf-4ec0-932d-73a0ec64030e","Type":"ContainerStarted","Data":"4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed"}
Oct 31 18:48:33.919365 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.919140    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" event={"ID":"96d452ef-90cf-4ec0-932d-73a0ec64030e","Type":"ContainerStarted","Data":"8c941ced82a2697d8814b34fe8ff773b00f345e50b881843e4c722c3061b4876"}
Oct 31 18:48:33.922012 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.921985    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7557-5a8696" event={"ID":"a0496eb0-bb46-48dc-be20-b3345323f146","Type":"ContainerStarted","Data":"d9e1864a602977a52a2900be92ffd3e4c2917acd0f4b6e3ea320bee5cda74851"}
Oct 31 18:48:33.922152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:33.922024    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7557-5a8696" event={"ID":"a0496eb0-bb46-48dc-be20-b3345323f146","Type":"ContainerStarted","Data":"46a9cf36e0edcd9a13600ee6d258177dde624fc6e99d6fd5add1fb1056842027"}
Oct 31 18:48:34.102578 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:34.102503    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:48:34.115221 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:34.115173    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7557-5a8696" podStartSLOduration=2.115158896 podStartE2EDuration="2.115158896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:34.112531173 +0000 UTC m=+1660.258981039" watchObservedRunningTime="2026-10-31 18:48:34.115158896 +0000 UTC m=+1660.261608761"
Oct 31 18:48:34.242494 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:34.240997    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" podStartSLOduration=2.240982214 podStartE2EDuration="2.240982214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:34.237786341 +0000 UTC m=+1660.384236207" watchObservedRunningTime="2026-10-31 18:48:34.240982214 +0000 UTC m=+1660.387432078"
Oct 31 18:48:34.382240 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:34.382162    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:34.383714 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:34.383690    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 18:48:34.383846 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:34.383744    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" podUID="96d452ef-90cf-4ec0-932d-73a0ec64030e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 18:48:35.001181 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.001144    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:35.226973 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.226935    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:35.385633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.383879    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 18:48:35.385633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.383937    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" podUID="96d452ef-90cf-4ec0-932d-73a0ec64030e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 18:48:35.550494 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.549815    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"]
Oct 31 18:48:35.575500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.564791    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"
Oct 31 18:48:35.621547 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.620935    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:35.621693 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.621566    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5dc7b6-dockercfg-4fmks\""
Oct 31 18:48:35.621953 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.621905    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5dc7b6\""
Oct 31 18:48:35.622571 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.622553    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"]
Oct 31 18:48:35.656907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.656691    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9cq\" (UniqueName: \"kubernetes.io/projected/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-kube-api-access-wq9cq\") pod \"payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz\" (UID: \"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"
Oct 31 18:48:35.656907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.656806    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz\" (UID: \"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"
Oct 31 18:48:35.758336 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.758303    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9cq\" (UniqueName: \"kubernetes.io/projected/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-kube-api-access-wq9cq\") pod \"payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz\" (UID: \"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"
Oct 31 18:48:35.758566 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.758373    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz\" (UID: \"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"
Oct 31 18:48:35.759108 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.759081    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz\" (UID: \"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"
Oct 31 18:48:35.816315 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.816285    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9cq\" (UniqueName: \"kubernetes.io/projected/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-kube-api-access-wq9cq\") pod \"payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz\" (UID: \"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"
Oct 31 18:48:35.841068 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.841039    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7557-5a8696"]
Oct 31 18:48:35.845221 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.845189    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7557-5a8696"]
Oct 31 18:48:35.935516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.935402    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7557-5a8696" podUID="a0496eb0-bb46-48dc-be20-b3345323f146" containerName="test-curl-7557-5a8696" containerID="cri-o://d9e1864a602977a52a2900be92ffd3e4c2917acd0f4b6e3ea320bee5cda74851" gracePeriod=2
Oct 31 18:48:35.956509 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:35.956462    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"
Oct 31 18:48:35.992502 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:48:35.991744    4484 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7557-5a8696\" is forbidden: User \"system:node:ip-10-0-142-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-120.us-west-1.compute.internal' and this object" podUID="a0496eb0-bb46-48dc-be20-b3345323f146" pod="opendatahub/test-curl-7557-5a8696"
Oct 31 18:48:36.383524 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:36.383460    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 18:48:36.384068 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:36.383550    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" podUID="96d452ef-90cf-4ec0-932d-73a0ec64030e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 18:48:36.407510 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:36.405651    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:36.418510 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:48:36.417884    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb5a4ed_960e_4e5d_86f1_5994a059c4b5.slice/crio-cd75d07225a7e13ae376a4abb8cb89d0dfa8940c105d800cdb0d23ad1d64bfb0 WatchSource:0}: Error finding container cd75d07225a7e13ae376a4abb8cb89d0dfa8940c105d800cdb0d23ad1d64bfb0: Status 404 returned error can't find the container with id cd75d07225a7e13ae376a4abb8cb89d0dfa8940c105d800cdb0d23ad1d64bfb0
Oct 31 18:48:36.418510 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:36.418394    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"]
Oct 31 18:48:36.943892 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:36.943859    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz" event={"ID":"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5","Type":"ContainerStarted","Data":"a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba"}
Oct 31 18:48:36.944084 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:36.943917    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"
Oct 31 18:48:36.944084 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:36.943934    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz" event={"ID":"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5","Type":"ContainerStarted","Data":"cd75d07225a7e13ae376a4abb8cb89d0dfa8940c105d800cdb0d23ad1d64bfb0"}
Oct 31 18:48:37.078502 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:37.061127    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz" podStartSLOduration=2.061107756 podStartE2EDuration="2.061107756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:37.058918593 +0000 UTC m=+1663.205368458" watchObservedRunningTime="2026-10-31 18:48:37.061107756 +0000 UTC m=+1663.207557622"
Oct 31 18:48:37.384500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:37.383952    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 18:48:37.384500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:37.384038    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" podUID="96d452ef-90cf-4ec0-932d-73a0ec64030e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 18:48:37.953383 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:37.953355    4484 generic.go:356] "Generic (PLEG): container finished" podID="a0496eb0-bb46-48dc-be20-b3345323f146" containerID="d9e1864a602977a52a2900be92ffd3e4c2917acd0f4b6e3ea320bee5cda74851" exitCode=137
Oct 31 18:48:38.231841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:38.231781    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7557-5a8696"
Oct 31 18:48:38.340539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:38.340504    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0496eb0-bb46-48dc-be20-b3345323f146-kube-api-access-ng7rc\" (UniqueName: \"kubernetes.io/projected/a0496eb0-bb46-48dc-be20-b3345323f146-kube-api-access-ng7rc\") pod \"a0496eb0-bb46-48dc-be20-b3345323f146\" (UID: \"a0496eb0-bb46-48dc-be20-b3345323f146\") "
Oct 31 18:48:38.342748 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:38.342716    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0496eb0-bb46-48dc-be20-b3345323f146-kube-api-access-ng7rc" pod "a0496eb0-bb46-48dc-be20-b3345323f146" (UID: "a0496eb0-bb46-48dc-be20-b3345323f146"). InnerVolumeSpecName "kube-api-access-ng7rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:38.385497 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:38.384864    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 18:48:38.385497 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:38.384925    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" podUID="96d452ef-90cf-4ec0-932d-73a0ec64030e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 18:48:38.441923 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:38.441891    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng7rc\" (UniqueName: \"kubernetes.io/projected/a0496eb0-bb46-48dc-be20-b3345323f146-kube-api-access-ng7rc\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:39.010501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:39.009842    4484 scope.go:122] "RemoveContainer" containerID="d9e1864a602977a52a2900be92ffd3e4c2917acd0f4b6e3ea320bee5cda74851"
Oct 31 18:48:39.010501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:39.009911    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7557-5a8696"
Oct 31 18:48:39.082773 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:48:39.082725    4484 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7557-5a8696\" is forbidden: User \"system:node:ip-10-0-142-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-120.us-west-1.compute.internal' and this object" podUID="a0496eb0-bb46-48dc-be20-b3345323f146" pod="opendatahub/test-curl-7557-5a8696"
Oct 31 18:48:39.383551 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:39.382872    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 18:48:39.383551 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:39.382934    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" podUID="96d452ef-90cf-4ec0-932d-73a0ec64030e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 18:48:39.999644 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:39.999613    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0496eb0-bb46-48dc-be20-b3345323f146" path="/var/lib/kubelet/pods/a0496eb0-bb46-48dc-be20-b3345323f146/volumes"
Oct 31 18:48:40.417929 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:40.417841    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:40.418368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:40.418311    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:40.419394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:40.419375    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:48:40.549697 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:40.549661    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:41.341215 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:41.341186    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"]
Oct 31 18:48:41.424296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:41.424259    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:44.619099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:44.617529    4484 patch_prober.go:29] interesting pod/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": read tcp 10.133.0.2:34686->10.133.0.114:15021: read: connection reset by peer" start-of-body=
Oct 31 18:48:44.619099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:44.617608    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": read tcp 10.133.0.2:34686->10.133.0.114:15021: read: connection reset by peer"
Oct 31 18:48:45.048946 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:45.048914    4484 generic.go:356] "Generic (PLEG): container finished" podID="0159e4a4-ad64-49f0-9256-9922e333235e" containerID="b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5" exitCode=0
Oct 31 18:48:45.049145 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:45.048969    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" event={"ID":"0159e4a4-ad64-49f0-9256-9922e333235e","Type":"ContainerDied","Data":"b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5"}
Oct 31 18:48:45.049429 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:45.049410    4484 scope.go:122] "RemoveContainer" containerID="b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5"
Oct 31 18:48:45.050871 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:45.050845    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:48:45.050939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:45.050910    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:48:45.050980 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:45.050941    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:48:45.174816 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:45.174782    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:46.057068 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:46.057039    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" event={"ID":"0159e4a4-ad64-49f0-9256-9922e333235e","Type":"ContainerStarted","Data":"238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a"}
Oct 31 18:48:46.750750 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:46.750720    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:48:46.751005 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:46.750984    4484 patch_prober.go:29] interesting pod/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 31 18:48:46.751050 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:46.751033    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 31 18:48:47.750773 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.750744    4484 patch_prober.go:29] interesting pod/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 31 18:48:47.751121 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.750813    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 31 18:48:47.887455 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.887417    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"]
Oct 31 18:48:47.888491 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.888450    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a0496eb0-bb46-48dc-be20-b3345323f146" containerName="test-curl-7557-5a8696"
Oct 31 18:48:47.888552 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.888495    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0496eb0-bb46-48dc-be20-b3345323f146" containerName="test-curl-7557-5a8696"
Oct 31 18:48:47.888662 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.888649    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0496eb0-bb46-48dc-be20-b3345323f146" containerName="test-curl-7557-5a8696"
Oct 31 18:48:47.892500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.892466    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:47.908059 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.908037    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:47.908179 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.908072    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:47.908179 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.908086    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:47.908253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.908218    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbdbfd3-cb79-4750-bf63-62622c589d4a-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:47.908253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.908248    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tblws\" (UniqueName: \"kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-kube-api-access-tblws\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:47.908338 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.908289    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:47.908400 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.908387    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:47.908445 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.908408    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:47.908524 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.908511    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:47.956300 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.956280    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"
Oct 31 18:48:47.988939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:47.988912    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"]
Oct 31 18:48:48.009946 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.009908    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.010312 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.010289    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.010599 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.010569    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.010760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.010734    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.010964 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.010944    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.011114 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.011097    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.011227 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.011192    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-data\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.011315 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.011223    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.011517 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.011449    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbdbfd3-cb79-4750-bf63-62622c589d4a-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.011628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.011534    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tblws\" (UniqueName: \"kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-kube-api-access-tblws\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.011628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.011536    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-credential-socket\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.011628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.011592    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.012023 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.012001    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbdbfd3-cb79-4750-bf63-62622c589d4a-istiod-ca-cert\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.012112 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.012068    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-certs\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.015000 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.014972    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-podinfo\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.015181 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.015159    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-envoy\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.088731 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.088697    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-token\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.088954 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.088937    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tblws\" (UniqueName: \"kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-kube-api-access-tblws\") pod \"e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") " pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.206643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.206615    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:48.534520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.534453    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:48.656191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.656074    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"]
Oct 31 18:48:48.656191 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:48:48.656168    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbdbfd3_cb79_4750_bf63_62622c589d4a.slice/crio-970c3337410586371188d3fe21d3b975d66a64fdaf48017b88fcbf8ef1a2d6cf WatchSource:0}: Error finding container 970c3337410586371188d3fe21d3b975d66a64fdaf48017b88fcbf8ef1a2d6cf: Status 404 returned error can't find the container with id 970c3337410586371188d3fe21d3b975d66a64fdaf48017b88fcbf8ef1a2d6cf
Oct 31 18:48:48.658940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.658906    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:48:48.659013 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.658971    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:48:48.659013 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.659001    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:48:48.751164 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.751136    4484 patch_prober.go:29] interesting pod/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 31 18:48:48.751560 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:48.751213    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 31 18:48:49.145810 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:49.145633    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:49.154432 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:49.154388    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" event={"ID":"adbdbfd3-cb79-4750-bf63-62622c589d4a","Type":"ContainerStarted","Data":"6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6"}
Oct 31 18:48:49.154691 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:49.154671    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" event={"ID":"adbdbfd3-cb79-4750-bf63-62622c589d4a","Type":"ContainerStarted","Data":"970c3337410586371188d3fe21d3b975d66a64fdaf48017b88fcbf8ef1a2d6cf"}
Oct 31 18:48:49.237137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:49.233372    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:49.237137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:49.235621    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 18:48:49.237137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:49.235687    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" podUID="adbdbfd3-cb79-4750-bf63-62622c589d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 18:48:49.632945 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:49.632904    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:49.750257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:49.750223    4484 patch_prober.go:29] interesting pod/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 31 18:48:49.750425 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:49.750272    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 31 18:48:49.842023 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:49.841988    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:50.015501 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:50.014057    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:50.216520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:50.215789    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 18:48:50.216520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:50.215847    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" podUID="adbdbfd3-cb79-4750-bf63-62622c589d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 18:48:50.750178 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:50.750145    4484 patch_prober.go:29] interesting pod/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 31 18:48:50.750360 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:50.750198    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 31 18:48:51.207675 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:51.207583    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 18:48:51.207675 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:51.207646    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" podUID="adbdbfd3-cb79-4750-bf63-62622c589d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 18:48:51.755040 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:51.755012    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:48:51.755446 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:51.755423    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:48:51.756206 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:51.756186    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:48:51.923416 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:51.923361    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" podStartSLOduration=4.923346561 podStartE2EDuration="4.923346561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:49.299109185 +0000 UTC m=+1675.445559052" watchObservedRunningTime="2026-10-31 18:48:51.923346561 +0000 UTC m=+1678.069796428"
Oct 31 18:48:52.207767 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:52.207690    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 18:48:52.207767 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:52.207748    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" podUID="adbdbfd3-cb79-4750-bf63-62622c589d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 18:48:53.187405 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:53.187367    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:53.207767 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:53.207737    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 18:48:53.208135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:53.207800    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" podUID="adbdbfd3-cb79-4750-bf63-62622c589d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 18:48:53.386984 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:53.386939    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:54.118248 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:54.118212    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:54.207138 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:54.207112    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 18:48:54.207330 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:54.207167    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" podUID="adbdbfd3-cb79-4750-bf63-62622c589d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 18:48:54.946623 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:54.946534    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:55.262109 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:55.262066    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:55.262991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:55.262968    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:55.263741 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:55.263721    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:48:55.468344 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:48:55.468303    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:48:56.362560 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.362505    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc" podUID="e451c404-1138-48ef-928e-51aa8584f45d" containerName="main" containerID="cri-o://c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc" gracePeriod=45
Oct 31 18:48:56.433792 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.433764    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:56.647736 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.647656    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:48:56.704066 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.703694    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:48:56.852672 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.852641    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-model-cache\") pod \"e451c404-1138-48ef-928e-51aa8584f45d\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") "
Oct 31 18:48:56.852854 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.852684    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e451c404-1138-48ef-928e-51aa8584f45d-kube-api-access-pmgtj\" (UniqueName: \"kubernetes.io/projected/e451c404-1138-48ef-928e-51aa8584f45d-kube-api-access-pmgtj\") pod \"e451c404-1138-48ef-928e-51aa8584f45d\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") "
Oct 31 18:48:56.852854 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.852734    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e451c404-1138-48ef-928e-51aa8584f45d-tls-certs\" (UniqueName: \"kubernetes.io/secret/e451c404-1138-48ef-928e-51aa8584f45d-tls-certs\") pod \"e451c404-1138-48ef-928e-51aa8584f45d\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") "
Oct 31 18:48:56.852991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.852859    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-tmp-dir\") pod \"e451c404-1138-48ef-928e-51aa8584f45d\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") "
Oct 31 18:48:56.852991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.852892    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-model-cache" pod "e451c404-1138-48ef-928e-51aa8584f45d" (UID: "e451c404-1138-48ef-928e-51aa8584f45d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:56.852991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.852908    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-dshm\") pod \"e451c404-1138-48ef-928e-51aa8584f45d\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") "
Oct 31 18:48:56.853141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.852996    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-home\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-home\") pod \"e451c404-1138-48ef-928e-51aa8584f45d\" (UID: \"e451c404-1138-48ef-928e-51aa8584f45d\") "
Oct 31 18:48:56.853183 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.853149    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-tmp-dir" pod "e451c404-1138-48ef-928e-51aa8584f45d" (UID: "e451c404-1138-48ef-928e-51aa8584f45d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:56.853250 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.853230    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-home" pod "e451c404-1138-48ef-928e-51aa8584f45d" (UID: "e451c404-1138-48ef-928e-51aa8584f45d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:56.853647 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.853546    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:56.853647 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.853569    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:56.853647 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.853585    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:56.855084 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.855063    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-dshm" pod "e451c404-1138-48ef-928e-51aa8584f45d" (UID: "e451c404-1138-48ef-928e-51aa8584f45d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:48:56.855174 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.855105    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e451c404-1138-48ef-928e-51aa8584f45d-tls-certs" pod "e451c404-1138-48ef-928e-51aa8584f45d" (UID: "e451c404-1138-48ef-928e-51aa8584f45d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:48:56.855221 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.855203    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e451c404-1138-48ef-928e-51aa8584f45d-kube-api-access-pmgtj" pod "e451c404-1138-48ef-928e-51aa8584f45d" (UID: "e451c404-1138-48ef-928e-51aa8584f45d"). InnerVolumeSpecName "kube-api-access-pmgtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:48:56.982552 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.965463    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e451c404-1138-48ef-928e-51aa8584f45d-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:56.982805 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.982772    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmgtj\" (UniqueName: \"kubernetes.io/projected/e451c404-1138-48ef-928e-51aa8584f45d-kube-api-access-pmgtj\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:56.982805 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:56.982804    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e451c404-1138-48ef-928e-51aa8584f45d-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:48:57.001789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.001427    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"]
Oct 31 18:48:57.004162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.004125    4484 cpu_manager.go:414] "containerMap: removing container" podUID="e451c404-1138-48ef-928e-51aa8584f45d" containerName="main"
Oct 31 18:48:57.004162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.004161    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e451c404-1138-48ef-928e-51aa8584f45d" containerName="main"
Oct 31 18:48:57.004405 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.004321    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e451c404-1138-48ef-928e-51aa8584f45d" containerName="main"
Oct 31 18:48:57.008662 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.008635    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"
Oct 31 18:48:57.027764 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.027724    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"]
Oct 31 18:48:57.061667 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.061638    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2a4923\""
Oct 31 18:48:57.062998 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.062973    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2a4923-dockercfg-4j9vs\""
Oct 31 18:48:57.083626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.083590    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6\" (UID: \"8ab961d9-00ff-4e95-8386-7cc9f7e0789f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"
Oct 31 18:48:57.083773 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.083690    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k29v\" (UniqueName: \"kubernetes.io/projected/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-kube-api-access-7k29v\") pod \"payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6\" (UID: \"8ab961d9-00ff-4e95-8386-7cc9f7e0789f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"
Oct 31 18:48:57.184586 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.184557    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6\" (UID: \"8ab961d9-00ff-4e95-8386-7cc9f7e0789f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"
Oct 31 18:48:57.184756 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.184625    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k29v\" (UniqueName: \"kubernetes.io/projected/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-kube-api-access-7k29v\") pod \"payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6\" (UID: \"8ab961d9-00ff-4e95-8386-7cc9f7e0789f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"
Oct 31 18:48:57.185263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.185246    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6\" (UID: \"8ab961d9-00ff-4e95-8386-7cc9f7e0789f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"
Oct 31 18:48:57.243149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.243123    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k29v\" (UniqueName: \"kubernetes.io/projected/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-kube-api-access-7k29v\") pod \"payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6\" (UID: \"8ab961d9-00ff-4e95-8386-7cc9f7e0789f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"
Oct 31 18:48:57.248137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.248114    4484 generic.go:356] "Generic (PLEG): container finished" podID="e451c404-1138-48ef-928e-51aa8584f45d" containerID="c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc" exitCode=0
Oct 31 18:48:57.248277 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.248160    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"
Oct 31 18:48:57.248277 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.248201    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc" event={"ID":"e451c404-1138-48ef-928e-51aa8584f45d","Type":"ContainerDied","Data":"c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc"}
Oct 31 18:48:57.248277 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.248239    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc" event={"ID":"e451c404-1138-48ef-928e-51aa8584f45d","Type":"ContainerDied","Data":"7a65110b2bb0c4a0d4b6940b935ccfae9ddeecb58b7e0ea799740804aa509ff2"}
Oct 31 18:48:57.248277 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.248260    4484 scope.go:122] "RemoveContainer" containerID="c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc"
Oct 31 18:48:57.258578 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.258559    4484 scope.go:122] "RemoveContainer" containerID="c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc"
Oct 31 18:48:57.258872 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:48:57.258855    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc\": container with ID starting with c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc not found: ID does not exist" containerID="c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc"
Oct 31 18:48:57.258920 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.258880    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc"} err="failed to get container status \"c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc\": rpc error: code = NotFound desc = could not find container \"c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc\": container with ID starting with c61d5547fe5d1c8464801a8df72ccc4e925e1163fe591dd5fe6034c0f99d39dc not found: ID does not exist"
Oct 31 18:48:57.331694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.331661    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"
Oct 31 18:48:57.415541 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.415511    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"]
Oct 31 18:48:57.418777 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.418752    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc"]
Oct 31 18:48:57.656759 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.656673    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-3417b822/e2e-lifecycle-model-3417b822-kserve-5bfd84f6c7-k64sc" podUID="e451c404-1138-48ef-928e-51aa8584f45d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 18:48:57.707386 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.707361    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"]
Oct 31 18:48:57.708059 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:48:57.708038    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab961d9_00ff_4e95_8386_7cc9f7e0789f.slice/crio-42459ca0e1ba06c940d2c26f545fcb0998ba683b9495fd96b8cd39374ccdaed5 WatchSource:0}: Error finding container 42459ca0e1ba06c940d2c26f545fcb0998ba683b9495fd96b8cd39374ccdaed5: Status 404 returned error can't find the container with id 42459ca0e1ba06c940d2c26f545fcb0998ba683b9495fd96b8cd39374ccdaed5
Oct 31 18:48:57.995139 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:57.995110    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e451c404-1138-48ef-928e-51aa8584f45d" path="/var/lib/kubelet/pods/e451c404-1138-48ef-928e-51aa8584f45d/volumes"
Oct 31 18:48:58.258500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:58.258392    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6" event={"ID":"8ab961d9-00ff-4e95-8386-7cc9f7e0789f","Type":"ContainerStarted","Data":"7f9c7e88d4d592cd42168bd7e813fba371bba6813a82fc81d83f07792ac6cff0"}
Oct 31 18:48:58.258500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:58.258445    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6" event={"ID":"8ab961d9-00ff-4e95-8386-7cc9f7e0789f","Type":"ContainerStarted","Data":"42459ca0e1ba06c940d2c26f545fcb0998ba683b9495fd96b8cd39374ccdaed5"}
Oct 31 18:48:58.258739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:58.258527    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"
Oct 31 18:48:58.373316 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:48:58.373261    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6" podStartSLOduration=2.373248217 podStartE2EDuration="2.373248217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:48:58.372048444 +0000 UTC m=+1684.518498309" watchObservedRunningTime="2026-10-31 18:48:58.373248217 +0000 UTC m=+1684.519698121"
Oct 31 18:49:00.094925 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:00.094893    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:03.182157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:03.182118    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:04.832569 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:04.832533    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:05.259209 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:05.259180    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:09.268294 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:09.268267    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"
Oct 31 18:49:09.957818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:09.957780    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:10.877054 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:10.877019    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:11.752319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:11.752289    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"]
Oct 31 18:49:12.131735 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:12.131652    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:12.330150 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:12.330119    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:15.046900 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:15.046858    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:16.726154 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:16.726123    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:16.753980 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:16.753938    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4" podUID="45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac" containerName="payload-processing" containerID="cri-o://bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227" gracePeriod=25
Oct 31 18:49:17.018746 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.018724    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"
Oct 31 18:49:17.151989 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.151954    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-kube-api-access-zt2fd\" (UniqueName: \"kubernetes.io/projected/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-kube-api-access-zt2fd\") pod \"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac\" (UID: \"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac\") "
Oct 31 18:49:17.152184 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.152103    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-plugins-config-volume\") pod \"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac\" (UID: \"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac\") "
Oct 31 18:49:17.152698 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.152670    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-plugins-config-volume" pod "45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac" (UID: "45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:49:17.154197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.154167    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-kube-api-access-zt2fd" pod "45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac" (UID: "45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac"). InnerVolumeSpecName "kube-api-access-zt2fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:17.253667 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.253602    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt2fd\" (UniqueName: \"kubernetes.io/projected/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-kube-api-access-zt2fd\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:17.253667 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.253629    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:17.398253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.398218    4484 generic.go:356] "Generic (PLEG): container finished" podID="45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac" containerID="bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227" exitCode=0
Oct 31 18:49:17.398437 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.398272    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"
Oct 31 18:49:17.398437 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.398293    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4" event={"ID":"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac","Type":"ContainerDied","Data":"bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227"}
Oct 31 18:49:17.398437 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.398366    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4" event={"ID":"45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac","Type":"ContainerDied","Data":"3ea05d9fa16d5b99c8c3240153da26202600b20160cffd5d1974e9d7cf18a8f5"}
Oct 31 18:49:17.398437 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.398387    4484 scope.go:122] "RemoveContainer" containerID="bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227"
Oct 31 18:49:17.409533 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.409516    4484 scope.go:122] "RemoveContainer" containerID="bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227"
Oct 31 18:49:17.409777 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:49:17.409759    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227\": container with ID starting with bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227 not found: ID does not exist" containerID="bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227"
Oct 31 18:49:17.409857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.409783    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227"} err="failed to get container status \"bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227\": rpc error: code = NotFound desc = could not find container \"bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227\": container with ID starting with bbcdcbbeb5c591002113d3605fb3a810ad3b6b22ce34c73f312936c2cb798227 not found: ID does not exist"
Oct 31 18:49:17.565569 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.565459    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"]
Oct 31 18:49:17.568302 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.568279    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c21e9e-5b7575f689-rqgs4"]
Oct 31 18:49:17.615820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.615790    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"]
Oct 31 18:49:17.691412 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.691241    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"]
Oct 31 18:49:17.968149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:17.968066    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:18.013149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:18.013117    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac" path="/var/lib/kubelet/pods/45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac/volumes"
Oct 31 18:49:18.167432 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:18.167396    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:18.422844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:18.422807    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:18.887086 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:18.887054    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:20.099864 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:20.099835    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:20.251293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:20.251260    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:20.397681 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:20.397604    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:22.616660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:22.616619    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9" podUID="81333aa2-5037-4dd9-8fce-3f1cf2531072" containerName="payload-processing" containerID="cri-o://f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83" gracePeriod=25
Oct 31 18:49:22.692399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:22.692342    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8" podUID="cbdb1b51-def9-40b6-818c-3009e3969017" containerName="payload-pre-processing" containerID="cri-o://e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c" gracePeriod=25
Oct 31 18:49:22.977164 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:22.977133    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"
Oct 31 18:49:23.007452 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.007423    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"
Oct 31 18:49:23.030591 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.030559    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81333aa2-5037-4dd9-8fce-3f1cf2531072-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81333aa2-5037-4dd9-8fce-3f1cf2531072-plugins-config-volume\") pod \"81333aa2-5037-4dd9-8fce-3f1cf2531072\" (UID: \"81333aa2-5037-4dd9-8fce-3f1cf2531072\") "
Oct 31 18:49:23.030778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.030625    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdb1b51-def9-40b6-818c-3009e3969017-kube-api-access-ps52d\" (UniqueName: \"kubernetes.io/projected/cbdb1b51-def9-40b6-818c-3009e3969017-kube-api-access-ps52d\") pod \"cbdb1b51-def9-40b6-818c-3009e3969017\" (UID: \"cbdb1b51-def9-40b6-818c-3009e3969017\") "
Oct 31 18:49:23.030778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.030689    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81333aa2-5037-4dd9-8fce-3f1cf2531072-kube-api-access-cddvp\" (UniqueName: \"kubernetes.io/projected/81333aa2-5037-4dd9-8fce-3f1cf2531072-kube-api-access-cddvp\") pod \"81333aa2-5037-4dd9-8fce-3f1cf2531072\" (UID: \"81333aa2-5037-4dd9-8fce-3f1cf2531072\") "
Oct 31 18:49:23.030894 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.030862    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbdb1b51-def9-40b6-818c-3009e3969017-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdb1b51-def9-40b6-818c-3009e3969017-plugins-config-volume\") pod \"cbdb1b51-def9-40b6-818c-3009e3969017\" (UID: \"cbdb1b51-def9-40b6-818c-3009e3969017\") "
Oct 31 18:49:23.030982 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.030954    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81333aa2-5037-4dd9-8fce-3f1cf2531072-plugins-config-volume" pod "81333aa2-5037-4dd9-8fce-3f1cf2531072" (UID: "81333aa2-5037-4dd9-8fce-3f1cf2531072"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:49:23.031395 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.031373    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81333aa2-5037-4dd9-8fce-3f1cf2531072-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:23.031573 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.031548    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdb1b51-def9-40b6-818c-3009e3969017-plugins-config-volume" pod "cbdb1b51-def9-40b6-818c-3009e3969017" (UID: "cbdb1b51-def9-40b6-818c-3009e3969017"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:49:23.033044 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.033021    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdb1b51-def9-40b6-818c-3009e3969017-kube-api-access-ps52d" pod "cbdb1b51-def9-40b6-818c-3009e3969017" (UID: "cbdb1b51-def9-40b6-818c-3009e3969017"). InnerVolumeSpecName "kube-api-access-ps52d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:23.033138 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.033041    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81333aa2-5037-4dd9-8fce-3f1cf2531072-kube-api-access-cddvp" pod "81333aa2-5037-4dd9-8fce-3f1cf2531072" (UID: "81333aa2-5037-4dd9-8fce-3f1cf2531072"). InnerVolumeSpecName "kube-api-access-cddvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:23.132729 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.132693    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdb1b51-def9-40b6-818c-3009e3969017-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:23.132729 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.132723    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps52d\" (UniqueName: \"kubernetes.io/projected/cbdb1b51-def9-40b6-818c-3009e3969017-kube-api-access-ps52d\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:23.132729 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.132734    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cddvp\" (UniqueName: \"kubernetes.io/projected/81333aa2-5037-4dd9-8fce-3f1cf2531072-kube-api-access-cddvp\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:23.452974 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.452938    4484 generic.go:356] "Generic (PLEG): container finished" podID="cbdb1b51-def9-40b6-818c-3009e3969017" containerID="e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c" exitCode=0
Oct 31 18:49:23.453204 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.453041    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"
Oct 31 18:49:23.453204 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.453023    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8" event={"ID":"cbdb1b51-def9-40b6-818c-3009e3969017","Type":"ContainerDied","Data":"e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c"}
Oct 31 18:49:23.453204 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.453111    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8" event={"ID":"cbdb1b51-def9-40b6-818c-3009e3969017","Type":"ContainerDied","Data":"d7b9d77f5e35ac867cff1a88f0536ab2dc925c61681d99b504bc07ae123aca4c"}
Oct 31 18:49:23.453204 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.453143    4484 scope.go:122] "RemoveContainer" containerID="e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c"
Oct 31 18:49:23.454819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.454796    4484 generic.go:356] "Generic (PLEG): container finished" podID="81333aa2-5037-4dd9-8fce-3f1cf2531072" containerID="f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83" exitCode=0
Oct 31 18:49:23.454910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.454829    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"
Oct 31 18:49:23.454910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.454871    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9" event={"ID":"81333aa2-5037-4dd9-8fce-3f1cf2531072","Type":"ContainerDied","Data":"f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83"}
Oct 31 18:49:23.454989 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.454913    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9" event={"ID":"81333aa2-5037-4dd9-8fce-3f1cf2531072","Type":"ContainerDied","Data":"e0934e4d77128bf51486393cf10c376e269ad6a3ade89b42162e78f49b9c8d2b"}
Oct 31 18:49:23.464281 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.464246    4484 scope.go:122] "RemoveContainer" containerID="e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c"
Oct 31 18:49:23.464680 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:49:23.464652    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c\": container with ID starting with e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c not found: ID does not exist" containerID="e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c"
Oct 31 18:49:23.464774 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.464689    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c"} err="failed to get container status \"e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c\": rpc error: code = NotFound desc = could not find container \"e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c\": container with ID starting with e0c43048684a09760cc2c4bf2b843e11cb48571d1dee3367d99cc32aafa3133c not found: ID does not exist"
Oct 31 18:49:23.464774 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.464712    4484 scope.go:122] "RemoveContainer" containerID="f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83"
Oct 31 18:49:23.473711 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.473692    4484 scope.go:122] "RemoveContainer" containerID="f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83"
Oct 31 18:49:23.474001 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:49:23.473980    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83\": container with ID starting with f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83 not found: ID does not exist" containerID="f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83"
Oct 31 18:49:23.474061 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.474008    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83"} err="failed to get container status \"f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83\": rpc error: code = NotFound desc = could not find container \"f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83\": container with ID starting with f1c8f6c20b6ca93b0c76a091b615dcb355a50e507f7beafe8fbf22f917504d83 not found: ID does not exist"
Oct 31 18:49:23.619039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.619007    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"]
Oct 31 18:49:23.671108 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.671071    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3417b822-675447c5c5-842q8"]
Oct 31 18:49:23.731810 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.731736    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"]
Oct 31 18:49:23.735157 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.735139    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3417b822-5bc7d88dbf-hv5s9"]
Oct 31 18:49:23.995203 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.995173    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81333aa2-5037-4dd9-8fce-3f1cf2531072" path="/var/lib/kubelet/pods/81333aa2-5037-4dd9-8fce-3f1cf2531072/volumes"
Oct 31 18:49:23.995666 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:23.995649    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbdb1b51-def9-40b6-818c-3009e3969017" path="/var/lib/kubelet/pods/cbdb1b51-def9-40b6-818c-3009e3969017/volumes"
Oct 31 18:49:24.420172 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:24.420077    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:24.715192 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:24.715097    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"]
Oct 31 18:49:24.756180 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:24.756141    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:24.830176 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:24.830143    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"]
Oct 31 18:49:25.088837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:25.088804    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:25.237960 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:25.237921    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:25.424853 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:25.424759    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:25.486154 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:49:25.486122    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:49:27.713602 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:27.713568    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:49:29.716320 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:29.716281    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74" podUID="7b89d508-6c46-4f54-a672-7f3cf4a6d96c" containerName="payload-processing" containerID="cri-o://8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072" gracePeriod=25
Oct 31 18:49:29.830906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:29.830870    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr" podUID="4120c0da-3927-463c-a732-40aad4707232" containerName="payload-pre-processing" containerID="cri-o://d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9" gracePeriod=25
Oct 31 18:49:29.978380 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:29.978314    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"
Oct 31 18:49:30.039412 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.039376    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-kube-api-access-zcg7d\" (UniqueName: \"kubernetes.io/projected/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-kube-api-access-zcg7d\") pod \"7b89d508-6c46-4f54-a672-7f3cf4a6d96c\" (UID: \"7b89d508-6c46-4f54-a672-7f3cf4a6d96c\") "
Oct 31 18:49:30.039608 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.039590    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-plugins-config-volume\") pod \"7b89d508-6c46-4f54-a672-7f3cf4a6d96c\" (UID: \"7b89d508-6c46-4f54-a672-7f3cf4a6d96c\") "
Oct 31 18:49:30.040044 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.039997    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-plugins-config-volume" pod "7b89d508-6c46-4f54-a672-7f3cf4a6d96c" (UID: "7b89d508-6c46-4f54-a672-7f3cf4a6d96c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:49:30.051377 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.051088    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:30.051526 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.051405    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-kube-api-access-zcg7d" pod "7b89d508-6c46-4f54-a672-7f3cf4a6d96c" (UID: "7b89d508-6c46-4f54-a672-7f3cf4a6d96c"). InnerVolumeSpecName "kube-api-access-zcg7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:30.105554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.105529    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"
Oct 31 18:49:30.152293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.152266    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4120c0da-3927-463c-a732-40aad4707232-kube-api-access-bpj66\" (UniqueName: \"kubernetes.io/projected/4120c0da-3927-463c-a732-40aad4707232-kube-api-access-bpj66\") pod \"4120c0da-3927-463c-a732-40aad4707232\" (UID: \"4120c0da-3927-463c-a732-40aad4707232\") "
Oct 31 18:49:30.152428 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.152354    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4120c0da-3927-463c-a732-40aad4707232-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4120c0da-3927-463c-a732-40aad4707232-plugins-config-volume\") pod \"4120c0da-3927-463c-a732-40aad4707232\" (UID: \"4120c0da-3927-463c-a732-40aad4707232\") "
Oct 31 18:49:30.152768 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.152746    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4120c0da-3927-463c-a732-40aad4707232-plugins-config-volume" pod "4120c0da-3927-463c-a732-40aad4707232" (UID: "4120c0da-3927-463c-a732-40aad4707232"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:49:30.152846 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.152834    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcg7d\" (UniqueName: \"kubernetes.io/projected/7b89d508-6c46-4f54-a672-7f3cf4a6d96c-kube-api-access-zcg7d\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:30.152880 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.152849    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4120c0da-3927-463c-a732-40aad4707232-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:30.154500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.154468    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4120c0da-3927-463c-a732-40aad4707232-kube-api-access-bpj66" pod "4120c0da-3927-463c-a732-40aad4707232" (UID: "4120c0da-3927-463c-a732-40aad4707232"). InnerVolumeSpecName "kube-api-access-bpj66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:49:30.240172 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.240116    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:30.253891 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.253864    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpj66\" (UniqueName: \"kubernetes.io/projected/4120c0da-3927-463c-a732-40aad4707232-kube-api-access-bpj66\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:49:30.520902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.520810    4484 generic.go:356] "Generic (PLEG): container finished" podID="7b89d508-6c46-4f54-a672-7f3cf4a6d96c" containerID="8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072" exitCode=0
Oct 31 18:49:30.520902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.520876    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"
Oct 31 18:49:30.521124 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.520893    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74" event={"ID":"7b89d508-6c46-4f54-a672-7f3cf4a6d96c","Type":"ContainerDied","Data":"8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072"}
Oct 31 18:49:30.521124 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.520990    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74" event={"ID":"7b89d508-6c46-4f54-a672-7f3cf4a6d96c","Type":"ContainerDied","Data":"281ac219f83c803dd463517b17e1db2c71ae186e9868b4aaa69a8dcabee0e71c"}
Oct 31 18:49:30.521124 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.521019    4484 scope.go:122] "RemoveContainer" containerID="8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072"
Oct 31 18:49:30.522262 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.522241    4484 generic.go:356] "Generic (PLEG): container finished" podID="4120c0da-3927-463c-a732-40aad4707232" containerID="d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9" exitCode=0
Oct 31 18:49:30.522333 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.522285    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"
Oct 31 18:49:30.522333 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.522305    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr" event={"ID":"4120c0da-3927-463c-a732-40aad4707232","Type":"ContainerDied","Data":"d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9"}
Oct 31 18:49:30.522413 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.522340    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr" event={"ID":"4120c0da-3927-463c-a732-40aad4707232","Type":"ContainerDied","Data":"47bdc4c2c6403e70dee3bcbf96ebd153de1df570e07074a76ff5b8ce10cf2335"}
Oct 31 18:49:30.531374 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.531209    4484 scope.go:122] "RemoveContainer" containerID="8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072"
Oct 31 18:49:30.531658 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:49:30.531632    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072\": container with ID starting with 8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072 not found: ID does not exist" containerID="8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072"
Oct 31 18:49:30.531757 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.531667    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072"} err="failed to get container status \"8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072\": rpc error: code = NotFound desc = could not find container \"8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072\": container with ID starting with 8ea60b04f9bbb40035d4a43360a1ea49fcbb644ffc6e360b258251039b92c072 not found: ID does not exist"
Oct 31 18:49:30.531757 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.531691    4484 scope.go:122] "RemoveContainer" containerID="d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9"
Oct 31 18:49:30.540280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.540263    4484 scope.go:122] "RemoveContainer" containerID="d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9"
Oct 31 18:49:30.540579 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:49:30.540561    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9\": container with ID starting with d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9 not found: ID does not exist" containerID="d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9"
Oct 31 18:49:30.540652 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.540588    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9"} err="failed to get container status \"d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9\": rpc error: code = NotFound desc = could not find container \"d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9\": container with ID starting with d28ad1146c98613262857c397b7ebd2bbcd2c9417f569febea7eb4d70dfcf1c9 not found: ID does not exist"
Oct 31 18:49:30.686940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.686909    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"]
Oct 31 18:49:30.739006 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.738971    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5d1b3f-64c575666c-lzs74"]
Oct 31 18:49:30.803794 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.803711    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"]
Oct 31 18:49:30.806496 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:30.806457    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d1b3f-6cf884b495-gsmjr"]
Oct 31 18:49:31.994915 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:31.994882    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4120c0da-3927-463c-a732-40aad4707232" path="/var/lib/kubelet/pods/4120c0da-3927-463c-a732-40aad4707232/volumes"
Oct 31 18:49:31.995291 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:31.995246    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b89d508-6c46-4f54-a672-7f3cf4a6d96c" path="/var/lib/kubelet/pods/7b89d508-6c46-4f54-a672-7f3cf4a6d96c/volumes"
Oct 31 18:49:34.285919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:34.285880    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:35.991818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:35.991787    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:40.592522 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:40.592453    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:40.815524 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:40.815484    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:42.247199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:42.247169    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:42.627391 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:42.627291    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:43.395830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:43.395792    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:45.623713 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:45.623679    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:45.829612 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:45.829570    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:45.984325 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:45.984247    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:48.917961 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:48.917929    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:49.119306 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:49.119279    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:49.855895 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:49.855856    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:50.006841 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:50.006804    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:50.938888 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:50.938850    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:53.405909 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:53.405868    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:53.580929 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:53.580890    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:54.464670 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:54.464635    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:54.595085 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:54.595053    4484 scope.go:122] "RemoveContainer" containerID="4ff350d3ae5b345ad5b1583b1ac68137b5fc9eb86c26c7ac5d4642a0089aac4b"
Oct 31 18:49:55.468521 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:49:55.468453    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:49:55.622529 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:55.622461    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:56.021065 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:56.021035    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:49:58.874030 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:49:58.873992    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:00.201683 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:00.201644    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:02.133253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:02.133215    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:04.611865 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:04.611827    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:04.813807 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:04.813773    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:05.655426 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:05.655348    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:06.187440 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:06.187404    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:06.338986 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:06.338926    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:06.724415 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:06.724389    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"]
Oct 31 18:50:06.724889 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:06.724633    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" podUID="96d452ef-90cf-4ec0-932d-73a0ec64030e" containerName="istio-proxy" containerID="cri-o://4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed" gracePeriod=30
Oct 31 18:50:06.758605 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:06.758574    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:06.760606 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:06.760586    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"]
Oct 31 18:50:06.760818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:06.760796    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" podUID="adbdbfd3-cb79-4750-bf63-62622c589d4a" containerName="istio-proxy" containerID="cri-o://6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6" gracePeriod=30
Oct 31 18:50:06.763420 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:06.763396    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"]
Oct 31 18:50:06.763778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:06.763751    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh" podUID="5ab42d63-0089-4fdd-9d07-441ba7c11b80" containerName="istio-proxy" containerID="cri-o://25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49" gracePeriod=30
Oct 31 18:50:07.440623 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:07.440584    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:08.052877 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:08.052842    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:08.338343 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:08.338246    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:09.059205 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:09.059168    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:10.197094 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:10.197061    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:50:10.197622 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:10.197124    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh" podUID="5ab42d63-0089-4fdd-9d07-441ba7c11b80" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:50:10.265909 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:10.265878    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:50:10.266066 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:10.265944    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" podUID="adbdbfd3-cb79-4750-bf63-62622c589d4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:50:10.420250 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:10.420212    4484 patch_prober.go:29] interesting pod/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:50:10.420431 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:10.420267    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" podUID="96d452ef-90cf-4ec0-932d-73a0ec64030e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:50:12.042265 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.041920    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:50:12.078987 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.078961    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:50:12.084411 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.084383    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:12.097351 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.097328    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:50:12.110411 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.110373    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-socket\") pod \"96d452ef-90cf-4ec0-932d-73a0ec64030e\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") "
Oct 31 18:50:12.110631 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.110451    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-credential-socket\") pod \"96d452ef-90cf-4ec0-932d-73a0ec64030e\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") "
Oct 31 18:50:12.110631 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.110514    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d452ef-90cf-4ec0-932d-73a0ec64030e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d452ef-90cf-4ec0-932d-73a0ec64030e-istiod-ca-cert\") pod \"96d452ef-90cf-4ec0-932d-73a0ec64030e\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") "
Oct 31 18:50:12.110631 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.110545    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-podinfo\") pod \"96d452ef-90cf-4ec0-932d-73a0ec64030e\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") "
Oct 31 18:50:12.110844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.110663    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-token\" (UniqueName: \"kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-token\") pod \"96d452ef-90cf-4ec0-932d-73a0ec64030e\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") "
Oct 31 18:50:12.110844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.110686    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-socket" pod "96d452ef-90cf-4ec0-932d-73a0ec64030e" (UID: "96d452ef-90cf-4ec0-932d-73a0ec64030e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.110844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.110722    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-certs\") pod \"96d452ef-90cf-4ec0-932d-73a0ec64030e\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") "
Oct 31 18:50:12.110844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.110761    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-credential-socket" pod "96d452ef-90cf-4ec0-932d-73a0ec64030e" (UID: "96d452ef-90cf-4ec0-932d-73a0ec64030e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.110844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.110772    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-envoy\") pod \"96d452ef-90cf-4ec0-932d-73a0ec64030e\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") "
Oct 31 18:50:12.110844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.110807    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-data\") pod \"96d452ef-90cf-4ec0-932d-73a0ec64030e\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") "
Oct 31 18:50:12.110844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.110839    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-kube-api-access-t72sk\" (UniqueName: \"kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-kube-api-access-t72sk\") pod \"96d452ef-90cf-4ec0-932d-73a0ec64030e\" (UID: \"96d452ef-90cf-4ec0-932d-73a0ec64030e\") "
Oct 31 18:50:12.111154 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.110961    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-certs" pod "96d452ef-90cf-4ec0-932d-73a0ec64030e" (UID: "96d452ef-90cf-4ec0-932d-73a0ec64030e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.111154 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.111004    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d452ef-90cf-4ec0-932d-73a0ec64030e-istiod-ca-cert" pod "96d452ef-90cf-4ec0-932d-73a0ec64030e" (UID: "96d452ef-90cf-4ec0-932d-73a0ec64030e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:50:12.111154 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.111129    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-data" pod "96d452ef-90cf-4ec0-932d-73a0ec64030e" (UID: "96d452ef-90cf-4ec0-932d-73a0ec64030e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.111692 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.111668    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.111755 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.111699    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.111755 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.111715    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.111755 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.111729    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.111755 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.111743    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96d452ef-90cf-4ec0-932d-73a0ec64030e-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.113060 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.113030    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-token" pod "96d452ef-90cf-4ec0-932d-73a0ec64030e" (UID: "96d452ef-90cf-4ec0-932d-73a0ec64030e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:12.113162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.113064    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-kube-api-access-t72sk" pod "96d452ef-90cf-4ec0-932d-73a0ec64030e" (UID: "96d452ef-90cf-4ec0-932d-73a0ec64030e"). InnerVolumeSpecName "kube-api-access-t72sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:12.113367 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.113340    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-podinfo" pod "96d452ef-90cf-4ec0-932d-73a0ec64030e" (UID: "96d452ef-90cf-4ec0-932d-73a0ec64030e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:50:12.113367 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.113351    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-envoy" pod "96d452ef-90cf-4ec0-932d-73a0ec64030e" (UID: "96d452ef-90cf-4ec0-932d-73a0ec64030e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.213219 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213173    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-data\") pod \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") "
Oct 31 18:50:12.213422 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213226    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-kube-api-access-22gc7\" (UniqueName: \"kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-kube-api-access-22gc7\") pod \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") "
Oct 31 18:50:12.213422 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213271    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-token\" (UniqueName: \"kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-token\") pod \"adbdbfd3-cb79-4750-bf63-62622c589d4a\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") "
Oct 31 18:50:12.213422 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213341    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-token\" (UniqueName: \"kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-token\") pod \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") "
Oct 31 18:50:12.213422 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213393    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-envoy\") pod \"adbdbfd3-cb79-4750-bf63-62622c589d4a\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") "
Oct 31 18:50:12.213692 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213440    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-socket\") pod \"adbdbfd3-cb79-4750-bf63-62622c589d4a\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") "
Oct 31 18:50:12.213692 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213445    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-data" pod "5ab42d63-0089-4fdd-9d07-441ba7c11b80" (UID: "5ab42d63-0089-4fdd-9d07-441ba7c11b80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.213692 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213495    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-podinfo\") pod \"adbdbfd3-cb79-4750-bf63-62622c589d4a\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") "
Oct 31 18:50:12.213692 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213525    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adbdbfd3-cb79-4750-bf63-62622c589d4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbdbfd3-cb79-4750-bf63-62622c589d4a-istiod-ca-cert\") pod \"adbdbfd3-cb79-4750-bf63-62622c589d4a\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") "
Oct 31 18:50:12.213692 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213550    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-data\") pod \"adbdbfd3-cb79-4750-bf63-62622c589d4a\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") "
Oct 31 18:50:12.213692 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213584    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istiod-ca-cert\") pod \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") "
Oct 31 18:50:12.213692 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213616    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-envoy\") pod \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") "
Oct 31 18:50:12.213692 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213649    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-credential-socket\") pod \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") "
Oct 31 18:50:12.214107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213706    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-credential-socket\") pod \"adbdbfd3-cb79-4750-bf63-62622c589d4a\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") "
Oct 31 18:50:12.214107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213745    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-certs\") pod \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") "
Oct 31 18:50:12.214107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213802    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-certs\") pod \"adbdbfd3-cb79-4750-bf63-62622c589d4a\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") "
Oct 31 18:50:12.214107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213832    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-socket\") pod \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") "
Oct 31 18:50:12.214107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213875    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-podinfo\") pod \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\" (UID: \"5ab42d63-0089-4fdd-9d07-441ba7c11b80\") "
Oct 31 18:50:12.214107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.213973    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-kube-api-access-tblws\" (UniqueName: \"kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-kube-api-access-tblws\") pod \"adbdbfd3-cb79-4750-bf63-62622c589d4a\" (UID: \"adbdbfd3-cb79-4750-bf63-62622c589d4a\") "
Oct 31 18:50:12.214399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.214288    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-certs" pod "adbdbfd3-cb79-4750-bf63-62622c589d4a" (UID: "adbdbfd3-cb79-4750-bf63-62622c589d4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.214522 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.214461    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-credential-socket" pod "adbdbfd3-cb79-4750-bf63-62622c589d4a" (UID: "adbdbfd3-cb79-4750-bf63-62622c589d4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.214872 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.214686    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-certs" pod "5ab42d63-0089-4fdd-9d07-441ba7c11b80" (UID: "5ab42d63-0089-4fdd-9d07-441ba7c11b80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.214872 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.214795    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.214872 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.214804    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbdbfd3-cb79-4750-bf63-62622c589d4a-istiod-ca-cert" pod "adbdbfd3-cb79-4750-bf63-62622c589d4a" (UID: "adbdbfd3-cb79-4750-bf63-62622c589d4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:50:12.214872 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.214814    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.214872 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.214831    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.215191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.214910    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-socket" pod "5ab42d63-0089-4fdd-9d07-441ba7c11b80" (UID: "5ab42d63-0089-4fdd-9d07-441ba7c11b80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.216657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.215263    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.216657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.215294    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.216657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.215310    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.216657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.215324    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96d452ef-90cf-4ec0-932d-73a0ec64030e-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.216657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.215339    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t72sk\" (UniqueName: \"kubernetes.io/projected/96d452ef-90cf-4ec0-932d-73a0ec64030e-kube-api-access-t72sk\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.216657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.215829    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-credential-socket" pod "5ab42d63-0089-4fdd-9d07-441ba7c11b80" (UID: "5ab42d63-0089-4fdd-9d07-441ba7c11b80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.216657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.215988    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-data" pod "adbdbfd3-cb79-4750-bf63-62622c589d4a" (UID: "adbdbfd3-cb79-4750-bf63-62622c589d4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.216657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.216228    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istiod-ca-cert" pod "5ab42d63-0089-4fdd-9d07-441ba7c11b80" (UID: "5ab42d63-0089-4fdd-9d07-441ba7c11b80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:50:12.217141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.216861    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-kube-api-access-22gc7" pod "5ab42d63-0089-4fdd-9d07-441ba7c11b80" (UID: "5ab42d63-0089-4fdd-9d07-441ba7c11b80"). InnerVolumeSpecName "kube-api-access-22gc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:12.217141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.217121    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-socket" pod "adbdbfd3-cb79-4750-bf63-62622c589d4a" (UID: "adbdbfd3-cb79-4750-bf63-62622c589d4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.217900 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.217835    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-token" pod "adbdbfd3-cb79-4750-bf63-62622c589d4a" (UID: "adbdbfd3-cb79-4750-bf63-62622c589d4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:12.218098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.218079    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-token" pod "5ab42d63-0089-4fdd-9d07-441ba7c11b80" (UID: "5ab42d63-0089-4fdd-9d07-441ba7c11b80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:12.218217 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.218185    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-podinfo" pod "5ab42d63-0089-4fdd-9d07-441ba7c11b80" (UID: "5ab42d63-0089-4fdd-9d07-441ba7c11b80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:50:12.218343 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.218233    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-envoy" pod "adbdbfd3-cb79-4750-bf63-62622c589d4a" (UID: "adbdbfd3-cb79-4750-bf63-62622c589d4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.218637 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.218613    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-envoy" pod "5ab42d63-0089-4fdd-9d07-441ba7c11b80" (UID: "5ab42d63-0089-4fdd-9d07-441ba7c11b80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:12.219327 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.219307    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-podinfo" pod "adbdbfd3-cb79-4750-bf63-62622c589d4a" (UID: "adbdbfd3-cb79-4750-bf63-62622c589d4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:50:12.219511 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.219452    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-kube-api-access-tblws" pod "adbdbfd3-cb79-4750-bf63-62622c589d4a" (UID: "adbdbfd3-cb79-4750-bf63-62622c589d4a"). InnerVolumeSpecName "kube-api-access-tblws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316505    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22gc7\" (UniqueName: \"kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-kube-api-access-22gc7\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316541    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316557    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316571    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316584    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316597    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316610    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbdbfd3-cb79-4750-bf63-62622c589d4a-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316624    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbdbfd3-cb79-4750-bf63-62622c589d4a-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316638    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316652    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316665    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316678    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab42d63-0089-4fdd-9d07-441ba7c11b80-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316690    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab42d63-0089-4fdd-9d07-441ba7c11b80-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317133 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.316703    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tblws\" (UniqueName: \"kubernetes.io/projected/adbdbfd3-cb79-4750-bf63-62622c589d4a-kube-api-access-tblws\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:12.317957 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.317906    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:12.474956 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.474928    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:12.790829 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.790794    4484 generic.go:356] "Generic (PLEG): container finished" podID="96d452ef-90cf-4ec0-932d-73a0ec64030e" containerID="4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed" exitCode=0
Oct 31 18:50:12.790991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.790839    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"
Oct 31 18:50:12.790991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.790876    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" event={"ID":"96d452ef-90cf-4ec0-932d-73a0ec64030e","Type":"ContainerDied","Data":"4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed"}
Oct 31 18:50:12.790991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.790927    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft" event={"ID":"96d452ef-90cf-4ec0-932d-73a0ec64030e","Type":"ContainerDied","Data":"8c941ced82a2697d8814b34fe8ff773b00f345e50b881843e4c722c3061b4876"}
Oct 31 18:50:12.790991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.790955    4484 scope.go:122] "RemoveContainer" containerID="4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed"
Oct 31 18:50:12.792385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.792348    4484 generic.go:356] "Generic (PLEG): container finished" podID="adbdbfd3-cb79-4750-bf63-62622c589d4a" containerID="6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6" exitCode=0
Oct 31 18:50:12.792761 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.792378    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" event={"ID":"adbdbfd3-cb79-4750-bf63-62622c589d4a","Type":"ContainerDied","Data":"6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6"}
Oct 31 18:50:12.792761 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.792398    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"
Oct 31 18:50:12.792761 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.792434    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9" event={"ID":"adbdbfd3-cb79-4750-bf63-62622c589d4a","Type":"ContainerDied","Data":"970c3337410586371188d3fe21d3b975d66a64fdaf48017b88fcbf8ef1a2d6cf"}
Oct 31 18:50:12.793966 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.793941    4484 generic.go:356] "Generic (PLEG): container finished" podID="5ab42d63-0089-4fdd-9d07-441ba7c11b80" containerID="25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49" exitCode=0
Oct 31 18:50:12.794085 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.794021    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"
Oct 31 18:50:12.794085 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.794053    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh" event={"ID":"5ab42d63-0089-4fdd-9d07-441ba7c11b80","Type":"ContainerDied","Data":"25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49"}
Oct 31 18:50:12.796782 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.794511    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh" event={"ID":"5ab42d63-0089-4fdd-9d07-441ba7c11b80","Type":"ContainerDied","Data":"2df4c0160f11e6822b190d86b10350411bef0010a53c6a79f49b958ce6872cfa"}
Oct 31 18:50:12.810725 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.810704    4484 scope.go:122] "RemoveContainer" containerID="4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed"
Oct 31 18:50:12.811049 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:50:12.811026    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed\": container with ID starting with 4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed not found: ID does not exist" containerID="4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed"
Oct 31 18:50:12.811149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.811054    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed"} err="failed to get container status \"4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed\": rpc error: code = NotFound desc = could not find container \"4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed\": container with ID starting with 4c090f95003a4c017e7fb6c82fa4cdff9875c3c829f6b51a6ea55fd77ce4b9ed not found: ID does not exist"
Oct 31 18:50:12.811149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.811072    4484 scope.go:122] "RemoveContainer" containerID="6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6"
Oct 31 18:50:12.820199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.820181    4484 scope.go:122] "RemoveContainer" containerID="6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6"
Oct 31 18:50:12.820591 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:50:12.820455    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6\": container with ID starting with 6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6 not found: ID does not exist" containerID="6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6"
Oct 31 18:50:12.820591 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.820515    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6"} err="failed to get container status \"6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6\": rpc error: code = NotFound desc = could not find container \"6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6\": container with ID starting with 6c9ab00fd72cd7b0d474116d1179db19645bb61b915367751bd1ec0803a594b6 not found: ID does not exist"
Oct 31 18:50:12.820591 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.820536    4484 scope.go:122] "RemoveContainer" containerID="25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49"
Oct 31 18:50:12.829689 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.829668    4484 scope.go:122] "RemoveContainer" containerID="25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49"
Oct 31 18:50:12.829949 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:50:12.829931    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49\": container with ID starting with 25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49 not found: ID does not exist" containerID="25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49"
Oct 31 18:50:12.830010 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.829956    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49"} err="failed to get container status \"25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49\": rpc error: code = NotFound desc = could not find container \"25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49\": container with ID starting with 25f98833a7a4a7a9ea0499cd020edb559de3040eac0dd5e438ed71095578ed49 not found: ID does not exist"
Oct 31 18:50:12.959193 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:12.959162    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"]
Oct 31 18:50:13.010005 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:13.009940    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-85knh"]
Oct 31 18:50:13.072896 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:13.072863    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"]
Oct 31 18:50:13.076123 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:13.076099    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-wb4c9"]
Oct 31 18:50:13.191572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:13.191504    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"]
Oct 31 18:50:13.195205 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:13.195184    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3417b822-openshift-default-7745fb7b8f-f8pft"]
Oct 31 18:50:13.994156 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:13.994127    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab42d63-0089-4fdd-9d07-441ba7c11b80" path="/var/lib/kubelet/pods/5ab42d63-0089-4fdd-9d07-441ba7c11b80/volumes"
Oct 31 18:50:13.994584 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:13.994571    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d452ef-90cf-4ec0-932d-73a0ec64030e" path="/var/lib/kubelet/pods/96d452ef-90cf-4ec0-932d-73a0ec64030e/volumes"
Oct 31 18:50:13.994997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:13.994984    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbdbfd3-cb79-4750-bf63-62622c589d4a" path="/var/lib/kubelet/pods/adbdbfd3-cb79-4750-bf63-62622c589d4a/volumes"
Oct 31 18:50:14.208077 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:14.208045    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:17.357052 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:17.357020    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:18.175694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:18.175653    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:18.796498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:18.794329    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:19.087027 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:19.086934    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:19.522892 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:19.522865    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:20.467606 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:20.467570    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:50:21.851361 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:21.851326    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:22.026260 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.026216    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:22.385630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.385418    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"]
Oct 31 18:50:22.386258 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386235    4484 cpu_manager.go:414] "containerMap: removing container" podUID="cbdb1b51-def9-40b6-818c-3009e3969017" containerName="payload-pre-processing"
Oct 31 18:50:22.386379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386261    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbdb1b51-def9-40b6-818c-3009e3969017" containerName="payload-pre-processing"
Oct 31 18:50:22.386379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386278    4484 cpu_manager.go:414] "containerMap: removing container" podUID="7b89d508-6c46-4f54-a672-7f3cf4a6d96c" containerName="payload-processing"
Oct 31 18:50:22.386379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386285    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b89d508-6c46-4f54-a672-7f3cf4a6d96c" containerName="payload-processing"
Oct 31 18:50:22.386379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386297    4484 cpu_manager.go:414] "containerMap: removing container" podUID="adbdbfd3-cb79-4750-bf63-62622c589d4a" containerName="istio-proxy"
Oct 31 18:50:22.386379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386304    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbdbfd3-cb79-4750-bf63-62622c589d4a" containerName="istio-proxy"
Oct 31 18:50:22.386379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386314    4484 cpu_manager.go:414] "containerMap: removing container" podUID="4120c0da-3927-463c-a732-40aad4707232" containerName="payload-pre-processing"
Oct 31 18:50:22.386379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386321    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4120c0da-3927-463c-a732-40aad4707232" containerName="payload-pre-processing"
Oct 31 18:50:22.386379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386341    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5ab42d63-0089-4fdd-9d07-441ba7c11b80" containerName="istio-proxy"
Oct 31 18:50:22.386379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386347    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab42d63-0089-4fdd-9d07-441ba7c11b80" containerName="istio-proxy"
Oct 31 18:50:22.386379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386358    4484 cpu_manager.go:414] "containerMap: removing container" podUID="45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac" containerName="payload-processing"
Oct 31 18:50:22.386379 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386365    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac" containerName="payload-processing"
Oct 31 18:50:22.386869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386392    4484 cpu_manager.go:414] "containerMap: removing container" podUID="96d452ef-90cf-4ec0-932d-73a0ec64030e" containerName="istio-proxy"
Oct 31 18:50:22.386869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386398    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d452ef-90cf-4ec0-932d-73a0ec64030e" containerName="istio-proxy"
Oct 31 18:50:22.386869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386436    4484 cpu_manager.go:414] "containerMap: removing container" podUID="81333aa2-5037-4dd9-8fce-3f1cf2531072" containerName="payload-processing"
Oct 31 18:50:22.386869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386442    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81333aa2-5037-4dd9-8fce-3f1cf2531072" containerName="payload-processing"
Oct 31 18:50:22.386869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386580    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab42d63-0089-4fdd-9d07-441ba7c11b80" containerName="istio-proxy"
Oct 31 18:50:22.386869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386594    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ef50d6-cfc1-4bb0-8597-ac7d5f8c7eac" containerName="payload-processing"
Oct 31 18:50:22.386869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386609    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d452ef-90cf-4ec0-932d-73a0ec64030e" containerName="istio-proxy"
Oct 31 18:50:22.386869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386622    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="81333aa2-5037-4dd9-8fce-3f1cf2531072" containerName="payload-processing"
Oct 31 18:50:22.386869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386630    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbdb1b51-def9-40b6-818c-3009e3969017" containerName="payload-pre-processing"
Oct 31 18:50:22.386869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386640    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b89d508-6c46-4f54-a672-7f3cf4a6d96c" containerName="payload-processing"
Oct 31 18:50:22.386869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386648    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbdbfd3-cb79-4750-bf63-62622c589d4a" containerName="istio-proxy"
Oct 31 18:50:22.386869 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.386656    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4120c0da-3927-463c-a732-40aad4707232" containerName="payload-pre-processing"
Oct 31 18:50:22.394972 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.394947    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"
Oct 31 18:50:22.424721 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.424688    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"]
Oct 31 18:50:22.442103 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.442061    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdvlq\" (UniqueName: \"kubernetes.io/projected/74824d04-008e-4e17-8ff6-1060a8a990a2-kube-api-access-xdvlq\") pod \"payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n\" (UID: \"74824d04-008e-4e17-8ff6-1060a8a990a2\") " pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"
Oct 31 18:50:22.442296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.442145    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74824d04-008e-4e17-8ff6-1060a8a990a2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n\" (UID: \"74824d04-008e-4e17-8ff6-1060a8a990a2\") " pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"
Oct 31 18:50:22.471100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.471067    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-aa4693\""
Oct 31 18:50:22.471355 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.471233    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-aa4693-dockercfg-28km8\""
Oct 31 18:50:22.543458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.543424    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvlq\" (UniqueName: \"kubernetes.io/projected/74824d04-008e-4e17-8ff6-1060a8a990a2-kube-api-access-xdvlq\") pod \"payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n\" (UID: \"74824d04-008e-4e17-8ff6-1060a8a990a2\") " pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"
Oct 31 18:50:22.543630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.543506    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74824d04-008e-4e17-8ff6-1060a8a990a2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n\" (UID: \"74824d04-008e-4e17-8ff6-1060a8a990a2\") " pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"
Oct 31 18:50:22.544517 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.544455    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74824d04-008e-4e17-8ff6-1060a8a990a2-plugins-config-volume\") pod \"payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n\" (UID: \"74824d04-008e-4e17-8ff6-1060a8a990a2\") " pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"
Oct 31 18:50:22.601410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.601382    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvlq\" (UniqueName: \"kubernetes.io/projected/74824d04-008e-4e17-8ff6-1060a8a990a2-kube-api-access-xdvlq\") pod \"payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n\" (UID: \"74824d04-008e-4e17-8ff6-1060a8a990a2\") " pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"
Oct 31 18:50:22.710654 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.710627    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"
Oct 31 18:50:22.877215 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:22.877187    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"]
Oct 31 18:50:23.107484 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:23.107445    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"]
Oct 31 18:50:23.108153 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:50:23.108128    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74824d04_008e_4e17_8ff6_1060a8a990a2.slice/crio-f065d7a09d4eeda5eb1d6889bd1ac7b4927e228a82385d021375ff02fde7d5ca WatchSource:0}: Error finding container f065d7a09d4eeda5eb1d6889bd1ac7b4927e228a82385d021375ff02fde7d5ca: Status 404 returned error can't find the container with id f065d7a09d4eeda5eb1d6889bd1ac7b4927e228a82385d021375ff02fde7d5ca
Oct 31 18:50:23.255877 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:23.255845    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:23.455605 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:23.455523    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:23.881154 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:23.881120    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n" event={"ID":"74824d04-008e-4e17-8ff6-1060a8a990a2","Type":"ContainerStarted","Data":"86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969"}
Oct 31 18:50:23.881619 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:23.881179    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n" event={"ID":"74824d04-008e-4e17-8ff6-1060a8a990a2","Type":"ContainerStarted","Data":"f065d7a09d4eeda5eb1d6889bd1ac7b4927e228a82385d021375ff02fde7d5ca"}
Oct 31 18:50:23.881619 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:23.881337    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"
Oct 31 18:50:23.998204 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:23.998147    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n" podStartSLOduration=1.998131844 podStartE2EDuration="1.998131844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:23.994742019 +0000 UTC m=+1770.141191915" watchObservedRunningTime="2026-10-31 18:50:23.998131844 +0000 UTC m=+1770.144581708"
Oct 31 18:50:24.486396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:24.486357    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:24.751943 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:24.751857    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:24.921393 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:24.921350    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:25.470304 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:50:25.470266    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:50:27.517092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:27.517054    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:27.877838 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:27.877752    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz" podUID="cbb5a4ed-960e-4e5d-86f1-5994a059c4b5" containerName="payload-processing" containerID="cri-o://a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba" gracePeriod=25
Oct 31 18:50:27.954895 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:50:27.954873    4484 logging.go:55] [core] [Channel #4304 SubChannel #4305]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.121:50052", ServerName: "10.133.0.121:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.121:50052: connect: connection refused"
Oct 31 18:50:28.006210 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.006184    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"]
Oct 31 18:50:28.010438 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.010418    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.129051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.128171    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-335e78-openshift-default-dockercfg-j7wpt\""
Oct 31 18:50:28.129051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.128700    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-data\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.129051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.128733    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2vm\" (UniqueName: \"kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-kube-api-access-tk2vm\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.129051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.128756    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-podinfo\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.129051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.128786    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-socket\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.129051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.128841    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-credential-socket\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.129051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.128866    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-certs\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.129051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.128893    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-envoy\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.129051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.128958    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istiod-ca-cert\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.129051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.129000    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-token\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.131673 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.131578    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:28.135806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.135784    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"]
Oct 31 18:50:28.230268 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.230234    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-certs\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.230421 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.230292    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-envoy\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.230421 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.230382    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istiod-ca-cert\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.230940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.230624    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-token\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.230940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.230727    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-data\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.230940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.230759    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2vm\" (UniqueName: \"kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-kube-api-access-tk2vm\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.230940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.230794    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-podinfo\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.230940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.230841    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-socket\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.230940 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.230880    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-certs\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.231272 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.230946    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-credential-socket\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.231666 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.231635    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-credential-socket\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.231784 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.231667    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-socket\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.231784 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.231765    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-data\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.232639 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.232614    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istiod-ca-cert\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.236520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.234260    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-envoy\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.236520 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.234907    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-podinfo\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.267584 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.267551    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"
Oct 31 18:50:28.288237 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.288172    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2vm\" (UniqueName: \"kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-kube-api-access-tk2vm\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.288937 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.288911    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-token\") pod \"e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") " pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.324134 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.324101    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:28.332166 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.332143    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-kube-api-access-wq9cq\" (UniqueName: \"kubernetes.io/projected/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-kube-api-access-wq9cq\") pod \"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5\" (UID: \"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5\") "
Oct 31 18:50:28.332342 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.332330    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-plugins-config-volume\") pod \"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5\" (UID: \"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5\") "
Oct 31 18:50:28.332704 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.332682    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-plugins-config-volume" pod "cbb5a4ed-960e-4e5d-86f1-5994a059c4b5" (UID: "cbb5a4ed-960e-4e5d-86f1-5994a059c4b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:50:28.334243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.334224    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-kube-api-access-wq9cq" pod "cbb5a4ed-960e-4e5d-86f1-5994a059c4b5" (UID: "cbb5a4ed-960e-4e5d-86f1-5994a059c4b5"). InnerVolumeSpecName "kube-api-access-wq9cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:28.428850 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.428814    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:28.434320 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.434295    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:28.434495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.434324    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9cq\" (UniqueName: \"kubernetes.io/projected/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5-kube-api-access-wq9cq\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:28.748232 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.748207    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"]
Oct 31 18:50:28.748658 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:50:28.748638    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d585fa_0b3c_4049_a004_c5fab33a5f2c.slice/crio-3a62192cd0f945d551ec35388b15f8af3dd6ab562bfe28631a2421865b8afdd2 WatchSource:0}: Error finding container 3a62192cd0f945d551ec35388b15f8af3dd6ab562bfe28631a2421865b8afdd2: Status 404 returned error can't find the container with id 3a62192cd0f945d551ec35388b15f8af3dd6ab562bfe28631a2421865b8afdd2
Oct 31 18:50:28.750932 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.750902    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:50:28.750997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.750965    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:50:28.751039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.750994    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:50:28.915024 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.914950    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" event={"ID":"c5d585fa-0b3c-4049-a004-c5fab33a5f2c","Type":"ContainerStarted","Data":"1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4"}
Oct 31 18:50:28.915211 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.915057    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" event={"ID":"c5d585fa-0b3c-4049-a004-c5fab33a5f2c","Type":"ContainerStarted","Data":"3a62192cd0f945d551ec35388b15f8af3dd6ab562bfe28631a2421865b8afdd2"}
Oct 31 18:50:28.916324 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.916300    4484 generic.go:356] "Generic (PLEG): container finished" podID="cbb5a4ed-960e-4e5d-86f1-5994a059c4b5" containerID="a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba" exitCode=0
Oct 31 18:50:28.916468 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.916338    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"
Oct 31 18:50:28.916468 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.916360    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz" event={"ID":"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5","Type":"ContainerDied","Data":"a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba"}
Oct 31 18:50:28.916468 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.916394    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz" event={"ID":"cbb5a4ed-960e-4e5d-86f1-5994a059c4b5","Type":"ContainerDied","Data":"cd75d07225a7e13ae376a4abb8cb89d0dfa8940c105d800cdb0d23ad1d64bfb0"}
Oct 31 18:50:28.916468 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.916414    4484 scope.go:122] "RemoveContainer" containerID="a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba"
Oct 31 18:50:28.926261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.926244    4484 scope.go:122] "RemoveContainer" containerID="a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba"
Oct 31 18:50:28.926585 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:50:28.926566    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba\": container with ID starting with a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba not found: ID does not exist" containerID="a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba"
Oct 31 18:50:28.926647 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.926593    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba"} err="failed to get container status \"a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba\": rpc error: code = NotFound desc = could not find container \"a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba\": container with ID starting with a1a1053e7008f342fb954f10f141ff297829e9fd803cbb7db80bb052bbcafcba not found: ID does not exist"
Oct 31 18:50:28.955372 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:28.955341    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz" podUID="cbb5a4ed-960e-4e5d-86f1-5994a059c4b5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.121:50052\" within 1s: context deadline exceeded"
Oct 31 18:50:28.955510 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:50:28.955384    4484 logging.go:55] [core] [Channel #4304 SubChannel #4305]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.121:50052", ServerName: "10.133.0.121:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.121:50052: operation was canceled"
Oct 31 18:50:29.105750 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:29.105663    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:29.153269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:29.153220    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" podStartSLOduration=2.15320495 podStartE2EDuration="2.15320495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:29.039390598 +0000 UTC m=+1775.185840462" watchObservedRunningTime="2026-10-31 18:50:29.15320495 +0000 UTC m=+1775.299654817"
Oct 31 18:50:29.153821 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:29.153797    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"]
Oct 31 18:50:29.162134 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:29.162099    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dc7b6-78db54cc79-h5pcz"]
Oct 31 18:50:29.309440 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:29.309407    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:29.324955 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:29.324934    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:29.330271 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:29.330250    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:29.577111 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:29.577078    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:29.924117 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:29.924046    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:29.925002 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:29.924986    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:50:29.993444 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:29.993418    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb5a4ed-960e-4e5d-86f1-5994a059c4b5" path="/var/lib/kubelet/pods/cbb5a4ed-960e-4e5d-86f1-5994a059c4b5/volumes"
Oct 31 18:50:31.817745 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:31.817702    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:32.131758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:32.131651    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:34.591384 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:34.591343    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:34.891124 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:34.891047    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"
Oct 31 18:50:36.199605 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.199566    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:36.363876 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.363840    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:36.585350 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.585317    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"]
Oct 31 18:50:36.585913 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.585900    4484 cpu_manager.go:414] "containerMap: removing container" podUID="cbb5a4ed-960e-4e5d-86f1-5994a059c4b5" containerName="payload-processing"
Oct 31 18:50:36.585954 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.585914    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb5a4ed-960e-4e5d-86f1-5994a059c4b5" containerName="payload-processing"
Oct 31 18:50:36.586019 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.586011    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb5a4ed-960e-4e5d-86f1-5994a059c4b5" containerName="payload-processing"
Oct 31 18:50:36.590755 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.590739    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"
Oct 31 18:50:36.709230 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.709175    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"]
Oct 31 18:50:36.736811 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.736782    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-335e78\""
Oct 31 18:50:36.737197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.737181    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-335e78-dockercfg-7g7hq\""
Oct 31 18:50:36.745605 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.745572    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l95v7\" (UniqueName: \"kubernetes.io/projected/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-kube-api-access-l95v7\") pod \"payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q\" (UID: \"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"
Oct 31 18:50:36.745875 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.745853    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q\" (UID: \"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"
Oct 31 18:50:36.847136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.847067    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l95v7\" (UniqueName: \"kubernetes.io/projected/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-kube-api-access-l95v7\") pod \"payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q\" (UID: \"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"
Oct 31 18:50:36.847136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.847123    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q\" (UID: \"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"
Oct 31 18:50:36.847809 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.847789    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q\" (UID: \"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"
Oct 31 18:50:36.904488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:36.904450    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l95v7\" (UniqueName: \"kubernetes.io/projected/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-kube-api-access-l95v7\") pod \"payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q\" (UID: \"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"
Oct 31 18:50:37.204416 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:37.204331    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"
Oct 31 18:50:37.579240 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:37.579208    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"]
Oct 31 18:50:37.579520 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:50:37.579491    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4deeea_bdb8_4e9c_bacd_cb0af59eba9b.slice/crio-3b389949093fb0f44fe74d12663b2bbbdf3d9221048b8365f9bfeb75636ba840 WatchSource:0}: Error finding container 3b389949093fb0f44fe74d12663b2bbbdf3d9221048b8365f9bfeb75636ba840: Status 404 returned error can't find the container with id 3b389949093fb0f44fe74d12663b2bbbdf3d9221048b8365f9bfeb75636ba840
Oct 31 18:50:37.972132 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:37.972047    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q" event={"ID":"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b","Type":"ContainerStarted","Data":"95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f"}
Oct 31 18:50:37.972132 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:37.972101    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q" event={"ID":"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b","Type":"ContainerStarted","Data":"3b389949093fb0f44fe74d12663b2bbbdf3d9221048b8365f9bfeb75636ba840"}
Oct 31 18:50:37.972322 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:37.972253    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"
Oct 31 18:50:38.094341 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:38.094290    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q" podStartSLOduration=2.094276935 podStartE2EDuration="2.094276935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:38.092505762 +0000 UTC m=+1784.238955627" watchObservedRunningTime="2026-10-31 18:50:38.094276935 +0000 UTC m=+1784.240726797"
Oct 31 18:50:38.706849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:38.706817    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:40.529769 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:40.529734    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:40.682179 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:40.682143    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:41.417728 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:41.417691    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:41.807975 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:41.807949    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"]
Oct 31 18:50:42.197221 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:42.197137    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:42.417229 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:42.417198    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:42.576083 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:42.576054    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:44.389162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:44.389125    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:45.667668 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:45.667636    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:45.889572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:45.889531    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:46.808965 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:46.808901    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6" podUID="8ab961d9-00ff-4e95-8386-7cc9f7e0789f" containerName="payload-processing" containerID="cri-o://7f9c7e88d4d592cd42168bd7e813fba371bba6813a82fc81d83f07792ac6cff0" gracePeriod=25
Oct 31 18:50:47.054818 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.054787    4484 generic.go:356] "Generic (PLEG): container finished" podID="8ab961d9-00ff-4e95-8386-7cc9f7e0789f" containerID="7f9c7e88d4d592cd42168bd7e813fba371bba6813a82fc81d83f07792ac6cff0" exitCode=0
Oct 31 18:50:47.054935 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.054858    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6" event={"ID":"8ab961d9-00ff-4e95-8386-7cc9f7e0789f","Type":"ContainerDied","Data":"7f9c7e88d4d592cd42168bd7e813fba371bba6813a82fc81d83f07792ac6cff0"}
Oct 31 18:50:47.083497 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.083460    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"
Oct 31 18:50:47.124905 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.124875    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-plugins-config-volume\") pod \"8ab961d9-00ff-4e95-8386-7cc9f7e0789f\" (UID: \"8ab961d9-00ff-4e95-8386-7cc9f7e0789f\") "
Oct 31 18:50:47.125107 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.125010    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-kube-api-access-7k29v\" (UniqueName: \"kubernetes.io/projected/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-kube-api-access-7k29v\") pod \"8ab961d9-00ff-4e95-8386-7cc9f7e0789f\" (UID: \"8ab961d9-00ff-4e95-8386-7cc9f7e0789f\") "
Oct 31 18:50:47.125318 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.125286    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-plugins-config-volume" pod "8ab961d9-00ff-4e95-8386-7cc9f7e0789f" (UID: "8ab961d9-00ff-4e95-8386-7cc9f7e0789f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:50:47.125442 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.125424    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:47.127213 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.127188    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-kube-api-access-7k29v" pod "8ab961d9-00ff-4e95-8386-7cc9f7e0789f" (UID: "8ab961d9-00ff-4e95-8386-7cc9f7e0789f"). InnerVolumeSpecName "kube-api-access-7k29v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:47.203936 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.203900    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:47.226365 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.226331    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k29v\" (UniqueName: \"kubernetes.io/projected/8ab961d9-00ff-4e95-8386-7cc9f7e0789f-kube-api-access-7k29v\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:47.654220 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.654176    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"]
Oct 31 18:50:47.654509 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.654442    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy" containerID="cri-o://238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a" gracePeriod=30
Oct 31 18:50:47.685783 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.685745    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:47.894077 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:47.894048    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:48.053980 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:48.053946    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:48.062292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:48.062270    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"
Oct 31 18:50:48.062292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:48.062283    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6" event={"ID":"8ab961d9-00ff-4e95-8386-7cc9f7e0789f","Type":"ContainerDied","Data":"42459ca0e1ba06c940d2c26f545fcb0998ba683b9495fd96b8cd39374ccdaed5"}
Oct 31 18:50:48.062511 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:48.062354    4484 scope.go:122] "RemoveContainer" containerID="7f9c7e88d4d592cd42168bd7e813fba371bba6813a82fc81d83f07792ac6cff0"
Oct 31 18:50:48.230490 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:48.230446    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"]
Oct 31 18:50:48.233408 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:48.233383    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a4923-78696ffb75-nk9z6"]
Oct 31 18:50:48.609792 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:48.609757    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:48.986153 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:48.986121    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"
Oct 31 18:50:49.233763 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:49.233728    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:49.813036 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:49.813004    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:49.993777 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:49.993748    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab961d9-00ff-4e95-8386-7cc9f7e0789f" path="/var/lib/kubelet/pods/8ab961d9-00ff-4e95-8386-7cc9f7e0789f/volumes"
Oct 31 18:50:50.355602 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:50.355570    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:50.994854 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:50.994826    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:51.158380 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.158349    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:51.308988 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.308909    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:51.382029 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.382001    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"]
Oct 31 18:50:51.382648 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.382632    4484 cpu_manager.go:414] "containerMap: removing container" podUID="8ab961d9-00ff-4e95-8386-7cc9f7e0789f" containerName="payload-processing"
Oct 31 18:50:51.382703 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.382649    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab961d9-00ff-4e95-8386-7cc9f7e0789f" containerName="payload-processing"
Oct 31 18:50:51.382753 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.382744    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab961d9-00ff-4e95-8386-7cc9f7e0789f" containerName="payload-processing"
Oct 31 18:50:51.387617 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.387596    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"
Oct 31 18:50:51.442764 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.442739    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"]
Oct 31 18:50:51.489742 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.489716    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ceef3b\""
Oct 31 18:50:51.540495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.528662    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htnww\" (UniqueName: \"kubernetes.io/projected/416036a1-61e2-4b66-92f7-3236cedbfcd4-kube-api-access-htnww\") pod \"payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v\" (UID: \"416036a1-61e2-4b66-92f7-3236cedbfcd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"
Oct 31 18:50:51.540495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.528778    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416036a1-61e2-4b66-92f7-3236cedbfcd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v\" (UID: \"416036a1-61e2-4b66-92f7-3236cedbfcd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"
Oct 31 18:50:51.567272 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.567196    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"]
Oct 31 18:50:51.571345 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.571322    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"
Oct 31 18:50:51.599005 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.598968    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"]
Oct 31 18:50:51.624671 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.624639    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ceef3b-dockercfg-qt7hs\""
Oct 31 18:50:51.630248 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.630221    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnww\" (UniqueName: \"kubernetes.io/projected/416036a1-61e2-4b66-92f7-3236cedbfcd4-kube-api-access-htnww\") pod \"payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v\" (UID: \"416036a1-61e2-4b66-92f7-3236cedbfcd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"
Oct 31 18:50:51.630378 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.630322    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416036a1-61e2-4b66-92f7-3236cedbfcd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v\" (UID: \"416036a1-61e2-4b66-92f7-3236cedbfcd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"
Oct 31 18:50:51.631251 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.631230    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416036a1-61e2-4b66-92f7-3236cedbfcd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v\" (UID: \"416036a1-61e2-4b66-92f7-3236cedbfcd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"
Oct 31 18:50:51.689207 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.689180    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnww\" (UniqueName: \"kubernetes.io/projected/416036a1-61e2-4b66-92f7-3236cedbfcd4-kube-api-access-htnww\") pod \"payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v\" (UID: \"416036a1-61e2-4b66-92f7-3236cedbfcd4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"
Oct 31 18:50:51.700437 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.700415    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"
Oct 31 18:50:51.731264 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.731227    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd6p7\" (UniqueName: \"kubernetes.io/projected/46cc80b3-2af6-442e-9080-8576b60ee024-kube-api-access-pd6p7\") pod \"payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch\" (UID: \"46cc80b3-2af6-442e-9080-8576b60ee024\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"
Oct 31 18:50:51.731537 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.731521    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46cc80b3-2af6-442e-9080-8576b60ee024-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch\" (UID: \"46cc80b3-2af6-442e-9080-8576b60ee024\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"
Oct 31 18:50:51.759561 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.759527    4484 patch_prober.go:29] interesting pod/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:50:51.759715 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.759586    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:50:51.833172 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.833093    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46cc80b3-2af6-442e-9080-8576b60ee024-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch\" (UID: \"46cc80b3-2af6-442e-9080-8576b60ee024\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"
Oct 31 18:50:51.833172 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.833149    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd6p7\" (UniqueName: \"kubernetes.io/projected/46cc80b3-2af6-442e-9080-8576b60ee024-kube-api-access-pd6p7\") pod \"payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch\" (UID: \"46cc80b3-2af6-442e-9080-8576b60ee024\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"
Oct 31 18:50:51.833845 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.833821    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46cc80b3-2af6-442e-9080-8576b60ee024-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch\" (UID: \"46cc80b3-2af6-442e-9080-8576b60ee024\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"
Oct 31 18:50:51.890632 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:51.890601    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd6p7\" (UniqueName: \"kubernetes.io/projected/46cc80b3-2af6-442e-9080-8576b60ee024-kube-api-access-pd6p7\") pod \"payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch\" (UID: \"46cc80b3-2af6-442e-9080-8576b60ee024\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"
Oct 31 18:50:52.090564 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:50:52.090525    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416036a1_61e2_4b66_92f7_3236cedbfcd4.slice/crio-5d9b65d3c4886e35c36f8c8722742c6e361dabd8743a056320b7fa3398745dd2 WatchSource:0}: Error finding container 5d9b65d3c4886e35c36f8c8722742c6e361dabd8743a056320b7fa3398745dd2: Status 404 returned error can't find the container with id 5d9b65d3c4886e35c36f8c8722742c6e361dabd8743a056320b7fa3398745dd2
Oct 31 18:50:52.090915 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:52.090625    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"]
Oct 31 18:50:52.120033 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:52.120007    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v" event={"ID":"416036a1-61e2-4b66-92f7-3236cedbfcd4","Type":"ContainerStarted","Data":"5d9b65d3c4886e35c36f8c8722742c6e361dabd8743a056320b7fa3398745dd2"}
Oct 31 18:50:52.188224 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:52.188197    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"
Oct 31 18:50:52.564192 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:52.564171    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"]
Oct 31 18:50:52.564746 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:50:52.564725    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cc80b3_2af6_442e_9080_8576b60ee024.slice/crio-46ec049b5a72567f19fb28cee6239cd8948fe372d0efeccaf1466d9b5cdfce26 WatchSource:0}: Error finding container 46ec049b5a72567f19fb28cee6239cd8948fe372d0efeccaf1466d9b5cdfce26: Status 404 returned error can't find the container with id 46ec049b5a72567f19fb28cee6239cd8948fe372d0efeccaf1466d9b5cdfce26
Oct 31 18:50:52.934363 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:52.934340    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:50:53.046648 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.046622    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-certs\") pod \"0159e4a4-ad64-49f0-9256-9922e333235e\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") "
Oct 31 18:50:53.046789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.046663    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-socket\") pod \"0159e4a4-ad64-49f0-9256-9922e333235e\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") "
Oct 31 18:50:53.046789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.046717    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-kube-api-access-mx52w\" (UniqueName: \"kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-kube-api-access-mx52w\") pod \"0159e4a4-ad64-49f0-9256-9922e333235e\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") "
Oct 31 18:50:53.046789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.046777    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0159e4a4-ad64-49f0-9256-9922e333235e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0159e4a4-ad64-49f0-9256-9922e333235e-istio-podinfo\") pod \"0159e4a4-ad64-49f0-9256-9922e333235e\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") "
Oct 31 18:50:53.046941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.046805    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-data\") pod \"0159e4a4-ad64-49f0-9256-9922e333235e\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") "
Oct 31 18:50:53.046941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.046850    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-envoy\") pod \"0159e4a4-ad64-49f0-9256-9922e333235e\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") "
Oct 31 18:50:53.046941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.046909    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-istio-token\" (UniqueName: \"kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-istio-token\") pod \"0159e4a4-ad64-49f0-9256-9922e333235e\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") "
Oct 31 18:50:53.047112 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.046955    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-credential-socket\") pod \"0159e4a4-ad64-49f0-9256-9922e333235e\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") "
Oct 31 18:50:53.047112 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.047003    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-socket" pod "0159e4a4-ad64-49f0-9256-9922e333235e" (UID: "0159e4a4-ad64-49f0-9256-9922e333235e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:53.047229 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.047107    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-data" pod "0159e4a4-ad64-49f0-9256-9922e333235e" (UID: "0159e4a4-ad64-49f0-9256-9922e333235e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:53.047229 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.047119    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0159e4a4-ad64-49f0-9256-9922e333235e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0159e4a4-ad64-49f0-9256-9922e333235e-istiod-ca-cert\") pod \"0159e4a4-ad64-49f0-9256-9922e333235e\" (UID: \"0159e4a4-ad64-49f0-9256-9922e333235e\") "
Oct 31 18:50:53.047337 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.047323    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-credential-socket" pod "0159e4a4-ad64-49f0-9256-9922e333235e" (UID: "0159e4a4-ad64-49f0-9256-9922e333235e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:53.047418 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.047390    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-certs" pod "0159e4a4-ad64-49f0-9256-9922e333235e" (UID: "0159e4a4-ad64-49f0-9256-9922e333235e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:53.047418 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.047407    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0159e4a4-ad64-49f0-9256-9922e333235e-istiod-ca-cert" pod "0159e4a4-ad64-49f0-9256-9922e333235e" (UID: "0159e4a4-ad64-49f0-9256-9922e333235e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:50:53.047744 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.047724    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:53.047744 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.047747    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:53.047921 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.047761    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0159e4a4-ad64-49f0-9256-9922e333235e-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:53.047921 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.047774    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:53.047921 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.047787    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:53.049168 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.049148    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0159e4a4-ad64-49f0-9256-9922e333235e-istio-podinfo" pod "0159e4a4-ad64-49f0-9256-9922e333235e" (UID: "0159e4a4-ad64-49f0-9256-9922e333235e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:50:53.049255 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.049163    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-envoy" pod "0159e4a4-ad64-49f0-9256-9922e333235e" (UID: "0159e4a4-ad64-49f0-9256-9922e333235e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:50:53.049611 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.049590    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-kube-api-access-mx52w" pod "0159e4a4-ad64-49f0-9256-9922e333235e" (UID: "0159e4a4-ad64-49f0-9256-9922e333235e"). InnerVolumeSpecName "kube-api-access-mx52w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:53.049611 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.049603    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-istio-token" pod "0159e4a4-ad64-49f0-9256-9922e333235e" (UID: "0159e4a4-ad64-49f0-9256-9922e333235e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:50:53.129438 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.129344    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v" event={"ID":"416036a1-61e2-4b66-92f7-3236cedbfcd4","Type":"ContainerStarted","Data":"dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e"}
Oct 31 18:50:53.129438 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.129414    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"
Oct 31 18:50:53.131441 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.131415    4484 generic.go:356] "Generic (PLEG): container finished" podID="0159e4a4-ad64-49f0-9256-9922e333235e" containerID="238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a" exitCode=0
Oct 31 18:50:53.131581 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.131461    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"
Oct 31 18:50:53.131633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.131579    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" event={"ID":"0159e4a4-ad64-49f0-9256-9922e333235e","Type":"ContainerDied","Data":"238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a"}
Oct 31 18:50:53.131670 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.131629    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv" event={"ID":"0159e4a4-ad64-49f0-9256-9922e333235e","Type":"ContainerDied","Data":"bf959b3d6a129769748d11dc9ca956f2035a9352565079dc7b817f51981516ce"}
Oct 31 18:50:53.131670 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.131659    4484 scope.go:122] "RemoveContainer" containerID="238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a"
Oct 31 18:50:53.140385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.133805    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch" event={"ID":"46cc80b3-2af6-442e-9080-8576b60ee024","Type":"ContainerStarted","Data":"32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7"}
Oct 31 18:50:53.140385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.133856    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch" event={"ID":"46cc80b3-2af6-442e-9080-8576b60ee024","Type":"ContainerStarted","Data":"46ec049b5a72567f19fb28cee6239cd8948fe372d0efeccaf1466d9b5cdfce26"}
Oct 31 18:50:53.140385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.134214    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"
Oct 31 18:50:53.145235 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.145216    4484 scope.go:122] "RemoveContainer" containerID="b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5"
Oct 31 18:50:53.150179 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.150158    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx52w\" (UniqueName: \"kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-kube-api-access-mx52w\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:53.150263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.150186    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0159e4a4-ad64-49f0-9256-9922e333235e-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:53.150263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.150205    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0159e4a4-ad64-49f0-9256-9922e333235e-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:53.150263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.150219    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0159e4a4-ad64-49f0-9256-9922e333235e-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:50:53.156603 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.156579    4484 scope.go:122] "RemoveContainer" containerID="238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a"
Oct 31 18:50:53.156867 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:50:53.156849    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a\": container with ID starting with 238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a not found: ID does not exist" containerID="238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a"
Oct 31 18:50:53.156925 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.156875    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a"} err="failed to get container status \"238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a\": rpc error: code = NotFound desc = could not find container \"238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a\": container with ID starting with 238dad0c786a443fac759fa90b01a9d3b5c634e45e2bf595bce87b72eeab4c3a not found: ID does not exist"
Oct 31 18:50:53.156925 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.156892    4484 scope.go:122] "RemoveContainer" containerID="b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5"
Oct 31 18:50:53.157188 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:50:53.157169    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5\": container with ID starting with b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5 not found: ID does not exist" containerID="b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5"
Oct 31 18:50:53.157264 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.157198    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5"} err="failed to get container status \"b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5\": rpc error: code = NotFound desc = could not find container \"b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5\": container with ID starting with b3cd9c30728e1e5d42c5ec4df92570f75b568a7ec632ac6d779227f6f4ccc0c5 not found: ID does not exist"
Oct 31 18:50:53.245708 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.245662    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v" podStartSLOduration=2.245649329 podStartE2EDuration="2.245649329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:53.242620566 +0000 UTC m=+1799.389070442" watchObservedRunningTime="2026-10-31 18:50:53.245649329 +0000 UTC m=+1799.392099190"
Oct 31 18:50:53.368316 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.365136    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"]
Oct 31 18:50:53.369582 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.369557    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-66032c-openshift-default-69cb4b8699-6pdzv"]
Oct 31 18:50:53.369951 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.369923    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:53.479102 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.479016    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch" podStartSLOduration=2.479003713 podStartE2EDuration="2.479003713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:50:53.478731796 +0000 UTC m=+1799.625181855" watchObservedRunningTime="2026-10-31 18:50:53.479003713 +0000 UTC m=+1799.625453575"
Oct 31 18:50:53.995943 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:53.995915    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" path="/var/lib/kubelet/pods/0159e4a4-ad64-49f0-9256-9922e333235e/volumes"
Oct 31 18:50:54.473459 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:54.473377    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:50:54.484424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:54.484403    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:50:54.789438 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:54.789408    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:55.468541 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:50:55.468503    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:50:58.069627 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.069590    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:58.226860 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.226823    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:58.373066 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.372980    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:50:58.456487 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.456451    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"]
Oct 31 18:50:58.457034 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.457022    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy"
Oct 31 18:50:58.457078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.457036    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy"
Oct 31 18:50:58.457169 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.457160    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy"
Oct 31 18:50:58.462202 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.462160    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"
Oct 31 18:50:58.534499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.527788    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"]
Oct 31 18:50:58.534499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.532109    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-956kb\" (UniqueName: \"kubernetes.io/projected/f62830c2-01ae-4f6d-acb2-5ab602d17919-kube-api-access-956kb\") pod \"payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7\" (UID: \"f62830c2-01ae-4f6d-acb2-5ab602d17919\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"
Oct 31 18:50:58.534499 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.532211    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62830c2-01ae-4f6d-acb2-5ab602d17919-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7\" (UID: \"f62830c2-01ae-4f6d-acb2-5ab602d17919\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"
Oct 31 18:50:58.576799 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.576763    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-676224-dockercfg-9rwzv\""
Oct 31 18:50:58.577202 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.577134    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-676224\""
Oct 31 18:50:58.630319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.630250    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"]
Oct 31 18:50:58.630832 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.630818    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy"
Oct 31 18:50:58.630874 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.630833    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy"
Oct 31 18:50:58.630966 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.630957    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0159e4a4-ad64-49f0-9256-9922e333235e" containerName="istio-proxy"
Oct 31 18:50:58.633856 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.633822    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62830c2-01ae-4f6d-acb2-5ab602d17919-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7\" (UID: \"f62830c2-01ae-4f6d-acb2-5ab602d17919\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"
Oct 31 18:50:58.634010 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.633995    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-956kb\" (UniqueName: \"kubernetes.io/projected/f62830c2-01ae-4f6d-acb2-5ab602d17919-kube-api-access-956kb\") pod \"payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7\" (UID: \"f62830c2-01ae-4f6d-acb2-5ab602d17919\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"
Oct 31 18:50:58.634341 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.634324    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"
Oct 31 18:50:58.634596 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.634577    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62830c2-01ae-4f6d-acb2-5ab602d17919-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7\" (UID: \"f62830c2-01ae-4f6d-acb2-5ab602d17919\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"
Oct 31 18:50:58.678373 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.678341    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"]
Oct 31 18:50:58.691704 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.691672    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-956kb\" (UniqueName: \"kubernetes.io/projected/f62830c2-01ae-4f6d-acb2-5ab602d17919-kube-api-access-956kb\") pod \"payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7\" (UID: \"f62830c2-01ae-4f6d-acb2-5ab602d17919\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"
Oct 31 18:50:58.734963 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.734933    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98t4\" (UniqueName: \"kubernetes.io/projected/ea5c4327-1827-43f0-8f1e-b711df9dacb6-kube-api-access-h98t4\") pod \"payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv\" (UID: \"ea5c4327-1827-43f0-8f1e-b711df9dacb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"
Oct 31 18:50:58.735092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.734973    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c4327-1827-43f0-8f1e-b711df9dacb6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv\" (UID: \"ea5c4327-1827-43f0-8f1e-b711df9dacb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"
Oct 31 18:50:58.776027 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.775999    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"
Oct 31 18:50:58.835835 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.835810    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98t4\" (UniqueName: \"kubernetes.io/projected/ea5c4327-1827-43f0-8f1e-b711df9dacb6-kube-api-access-h98t4\") pod \"payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv\" (UID: \"ea5c4327-1827-43f0-8f1e-b711df9dacb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"
Oct 31 18:50:58.835957 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.835855    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c4327-1827-43f0-8f1e-b711df9dacb6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv\" (UID: \"ea5c4327-1827-43f0-8f1e-b711df9dacb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"
Oct 31 18:50:58.836618 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.836596    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c4327-1827-43f0-8f1e-b711df9dacb6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv\" (UID: \"ea5c4327-1827-43f0-8f1e-b711df9dacb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"
Oct 31 18:50:58.893899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.893821    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98t4\" (UniqueName: \"kubernetes.io/projected/ea5c4327-1827-43f0-8f1e-b711df9dacb6-kube-api-access-h98t4\") pod \"payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv\" (UID: \"ea5c4327-1827-43f0-8f1e-b711df9dacb6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"
Oct 31 18:50:58.947964 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:58.947936    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"
Oct 31 18:50:59.151725 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:59.151693    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"]
Oct 31 18:50:59.152360 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:50:59.152340    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62830c2_01ae_4f6d_acb2_5ab602d17919.slice/crio-6c6ad61b3d6cabb7f57629c6b3088d4415d185b66b7119fe0c58e692f814c0c1 WatchSource:0}: Error finding container 6c6ad61b3d6cabb7f57629c6b3088d4415d185b66b7119fe0c58e692f814c0c1: Status 404 returned error can't find the container with id 6c6ad61b3d6cabb7f57629c6b3088d4415d185b66b7119fe0c58e692f814c0c1
Oct 31 18:50:59.175813 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:59.175780    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7" event={"ID":"f62830c2-01ae-4f6d-acb2-5ab602d17919","Type":"ContainerStarted","Data":"6c6ad61b3d6cabb7f57629c6b3088d4415d185b66b7119fe0c58e692f814c0c1"}
Oct 31 18:50:59.320628 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:50:59.320601    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"]
Oct 31 18:50:59.320734 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:50:59.320597    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5c4327_1827_43f0_8f1e_b711df9dacb6.slice/crio-67d213b0c625b7ab359a625ce409e31ed4395cf377c6c1a2b978b2d4985e37df WatchSource:0}: Error finding container 67d213b0c625b7ab359a625ce409e31ed4395cf377c6c1a2b978b2d4985e37df: Status 404 returned error can't find the container with id 67d213b0c625b7ab359a625ce409e31ed4395cf377c6c1a2b978b2d4985e37df
Oct 31 18:51:00.183736 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:00.183698    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv" event={"ID":"ea5c4327-1827-43f0-8f1e-b711df9dacb6","Type":"ContainerStarted","Data":"6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433"}
Oct 31 18:51:00.184189 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:00.183765    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv" event={"ID":"ea5c4327-1827-43f0-8f1e-b711df9dacb6","Type":"ContainerStarted","Data":"67d213b0c625b7ab359a625ce409e31ed4395cf377c6c1a2b978b2d4985e37df"}
Oct 31 18:51:00.184189 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:00.183797    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"
Oct 31 18:51:00.185349 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:00.185324    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7" event={"ID":"f62830c2-01ae-4f6d-acb2-5ab602d17919","Type":"ContainerStarted","Data":"345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e"}
Oct 31 18:51:00.185487 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:00.185368    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"
Oct 31 18:51:00.297837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:00.297777    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv" podStartSLOduration=2.29776479 podStartE2EDuration="2.29776479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:51:00.296979423 +0000 UTC m=+1806.443429288" watchObservedRunningTime="2026-10-31 18:51:00.29776479 +0000 UTC m=+1806.444214653"
Oct 31 18:51:00.412528 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:00.411658    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7" podStartSLOduration=2.41164065 podStartE2EDuration="2.41164065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:51:00.409710988 +0000 UTC m=+1806.556160843" watchObservedRunningTime="2026-10-31 18:51:00.41164065 +0000 UTC m=+1806.558090513"
Oct 31 18:51:01.965356 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:01.965323    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:02.491800 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:02.491762    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:03.658665 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:03.658631    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:51:04.144468 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:04.144443    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"
Oct 31 18:51:04.144669 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:04.144540    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"
Oct 31 18:51:05.271673 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:05.271642    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:05.430099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:05.430066    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:05.599313 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:05.599231    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:10.305129 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:10.305094    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"]
Oct 31 18:51:10.305534 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:10.305351    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg" podUID="bcd95837-2a87-4361-877a-2d2edfd66977" containerName="istio-proxy" containerID="cri-o://86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89" gracePeriod=30
Oct 31 18:51:10.455212 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:10.455176    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:11.194878 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:11.194828    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"
Oct 31 18:51:11.195061 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:11.194920    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"
Oct 31 18:51:15.205275 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.205241    4484 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:51:15.205716 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.205335    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg" podUID="bcd95837-2a87-4361-877a-2d2edfd66977" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:51:15.244294 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.244255    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:15.579308 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.579286    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:51:15.659871 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.659842    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-certs\") pod \"bcd95837-2a87-4361-877a-2d2edfd66977\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") "
Oct 31 18:51:15.659871 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.659885    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-credential-socket\") pod \"bcd95837-2a87-4361-877a-2d2edfd66977\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") "
Oct 31 18:51:15.660116 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.659961    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-kube-api-access-k6tmh\" (UniqueName: \"kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-kube-api-access-k6tmh\") pod \"bcd95837-2a87-4361-877a-2d2edfd66977\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") "
Oct 31 18:51:15.660116 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660048    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-data\") pod \"bcd95837-2a87-4361-877a-2d2edfd66977\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") "
Oct 31 18:51:15.660116 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660090    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-socket\") pod \"bcd95837-2a87-4361-877a-2d2edfd66977\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") "
Oct 31 18:51:15.660269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660144    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd95837-2a87-4361-877a-2d2edfd66977-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd95837-2a87-4361-877a-2d2edfd66977-istiod-ca-cert\") pod \"bcd95837-2a87-4361-877a-2d2edfd66977\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") "
Oct 31 18:51:15.660269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660149    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-certs" pod "bcd95837-2a87-4361-877a-2d2edfd66977" (UID: "bcd95837-2a87-4361-877a-2d2edfd66977"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:51:15.660269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660188    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-envoy\") pod \"bcd95837-2a87-4361-877a-2d2edfd66977\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") "
Oct 31 18:51:15.660269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660200    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-credential-socket" pod "bcd95837-2a87-4361-877a-2d2edfd66977" (UID: "bcd95837-2a87-4361-877a-2d2edfd66977"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:51:15.660269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660265    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcd95837-2a87-4361-877a-2d2edfd66977-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd95837-2a87-4361-877a-2d2edfd66977-istio-podinfo\") pod \"bcd95837-2a87-4361-877a-2d2edfd66977\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") "
Oct 31 18:51:15.660539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660300    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-istio-token\" (UniqueName: \"kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-istio-token\") pod \"bcd95837-2a87-4361-877a-2d2edfd66977\" (UID: \"bcd95837-2a87-4361-877a-2d2edfd66977\") "
Oct 31 18:51:15.660539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660345    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-data" pod "bcd95837-2a87-4361-877a-2d2edfd66977" (UID: "bcd95837-2a87-4361-877a-2d2edfd66977"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:51:15.660539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660356    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-socket" pod "bcd95837-2a87-4361-877a-2d2edfd66977" (UID: "bcd95837-2a87-4361-877a-2d2edfd66977"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:51:15.660539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660513    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd95837-2a87-4361-877a-2d2edfd66977-istiod-ca-cert" pod "bcd95837-2a87-4361-877a-2d2edfd66977" (UID: "bcd95837-2a87-4361-877a-2d2edfd66977"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:51:15.660833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660800    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:51:15.660833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660824    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:51:15.660939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660837    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:51:15.660939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660849    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:51:15.660939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.660862    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcd95837-2a87-4361-877a-2d2edfd66977-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:51:15.662365 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.662337    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-envoy" pod "bcd95837-2a87-4361-877a-2d2edfd66977" (UID: "bcd95837-2a87-4361-877a-2d2edfd66977"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:51:15.662540 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.662434    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcd95837-2a87-4361-877a-2d2edfd66977-istio-podinfo" pod "bcd95837-2a87-4361-877a-2d2edfd66977" (UID: "bcd95837-2a87-4361-877a-2d2edfd66977"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:51:15.662868 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.662849    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-kube-api-access-k6tmh" pod "bcd95837-2a87-4361-877a-2d2edfd66977" (UID: "bcd95837-2a87-4361-877a-2d2edfd66977"). InnerVolumeSpecName "kube-api-access-k6tmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:51:15.662968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.662866    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-istio-token" pod "bcd95837-2a87-4361-877a-2d2edfd66977" (UID: "bcd95837-2a87-4361-877a-2d2edfd66977"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:51:15.761762 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.761733    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcd95837-2a87-4361-877a-2d2edfd66977-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:51:15.761762 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.761760    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcd95837-2a87-4361-877a-2d2edfd66977-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:51:15.761932 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.761769    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:51:15.761932 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:15.761778    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6tmh\" (UniqueName: \"kubernetes.io/projected/bcd95837-2a87-4361-877a-2d2edfd66977-kube-api-access-k6tmh\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:51:16.308970 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:16.308936    4484 generic.go:356] "Generic (PLEG): container finished" podID="bcd95837-2a87-4361-877a-2d2edfd66977" containerID="86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89" exitCode=0
Oct 31 18:51:16.309372 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:16.309003    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"
Oct 31 18:51:16.309372 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:16.309014    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg" event={"ID":"bcd95837-2a87-4361-877a-2d2edfd66977","Type":"ContainerDied","Data":"86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89"}
Oct 31 18:51:16.309372 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:16.309098    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg" event={"ID":"bcd95837-2a87-4361-877a-2d2edfd66977","Type":"ContainerDied","Data":"1e5827af6253ffbf0f148c7c19a9a9c302876a4cfbcfbc262449bb54de065e6a"}
Oct 31 18:51:16.309372 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:16.309129    4484 scope.go:122] "RemoveContainer" containerID="86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89"
Oct 31 18:51:16.319027 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:16.319010    4484 scope.go:122] "RemoveContainer" containerID="86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89"
Oct 31 18:51:16.319303 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:51:16.319282    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89\": container with ID starting with 86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89 not found: ID does not exist" containerID="86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89"
Oct 31 18:51:16.319371 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:16.319316    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89"} err="failed to get container status \"86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89\": rpc error: code = NotFound desc = could not find container \"86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89\": container with ID starting with 86b9abde090449bbe70c8b72b04f24ae4aa1d16021aca0be5430f2e51a4a4c89 not found: ID does not exist"
Oct 31 18:51:16.526685 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:16.526653    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"]
Oct 31 18:51:16.529347 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:16.529323    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-5t9kg"]
Oct 31 18:51:17.993864 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:17.993834    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd95837-2a87-4361-877a-2d2edfd66977" path="/var/lib/kubelet/pods/bcd95837-2a87-4361-877a-2d2edfd66977/volumes"
Oct 31 18:51:19.591993 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.591962    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"]
Oct 31 18:51:19.592805 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.592783    4484 cpu_manager.go:414] "containerMap: removing container" podUID="bcd95837-2a87-4361-877a-2d2edfd66977" containerName="istio-proxy"
Oct 31 18:51:19.592805 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.592804    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd95837-2a87-4361-877a-2d2edfd66977" containerName="istio-proxy"
Oct 31 18:51:19.592985 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.592935    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd95837-2a87-4361-877a-2d2edfd66977" containerName="istio-proxy"
Oct 31 18:51:19.597635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.597615    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:19.650963 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.650926    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-100faf-serving-cert\""
Oct 31 18:51:19.702487 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.702452    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:19.702659 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.702578    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-metrics-certs\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:19.702659 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.702626    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ks4c\" (UniqueName: \"kubernetes.io/projected/bd67e094-084a-4c09-bc37-1ec4a40cc2da-kube-api-access-7ks4c\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:19.702800 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.702780    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd67e094-084a-4c09-bc37-1ec4a40cc2da-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:19.704744 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.704727    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"]
Oct 31 18:51:19.804313 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.804280    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:19.804516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.804337    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-metrics-certs\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:19.804516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.804362    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ks4c\" (UniqueName: \"kubernetes.io/projected/bd67e094-084a-4c09-bc37-1ec4a40cc2da-kube-api-access-7ks4c\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:19.804516 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:51:19.804428    4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-serving-cert: secret "maas-api-e2e-ipp-b-100faf-serving-cert" not found
Oct 31 18:51:19.804516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.804433    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd67e094-084a-4c09-bc37-1ec4a40cc2da-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:19.804691 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:51:19.804518    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-maas-api-tls podName:bd67e094-084a-4c09-bc37-1ec4a40cc2da nodeName:}" failed. No retries permitted until 2026-10-31 18:51:20.304496701 +0000 UTC m=+1826.450946568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-maas-api-tls") pod "maas-api-e2e-ipp-b-100faf-55658c47d6-h752r" (UID: "bd67e094-084a-4c09-bc37-1ec4a40cc2da") : secret "maas-api-e2e-ipp-b-100faf-serving-cert" not found
Oct 31 18:51:19.805010 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.804991    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd67e094-084a-4c09-bc37-1ec4a40cc2da-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:19.806952 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.806933    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-metrics-certs\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:19.816309 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.816215    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:19.862377 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.862309    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ks4c\" (UniqueName: \"kubernetes.io/projected/bd67e094-084a-4c09-bc37-1ec4a40cc2da-kube-api-access-7ks4c\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:19.992906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:19.992874    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:20.145823 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:20.145734    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:20.311702 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:20.311673    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:20.314402 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:20.314378    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-100faf-55658c47d6-h752r\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:20.510656 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:20.510624    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:20.886659 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:51:20.886628    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd67e094_084a_4c09_bc37_1ec4a40cc2da.slice/crio-6715c06e0f48b98e8a3b30617824b405f9d5954b9758518e62b43f03cd725f55 WatchSource:0}: Error finding container 6715c06e0f48b98e8a3b30617824b405f9d5954b9758518e62b43f03cd725f55: Status 404 returned error can't find the container with id 6715c06e0f48b98e8a3b30617824b405f9d5954b9758518e62b43f03cd725f55
Oct 31 18:51:20.887073 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:20.886670    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"]
Oct 31 18:51:21.340112 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:21.340076    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r" event={"ID":"bd67e094-084a-4c09-bc37-1ec4a40cc2da","Type":"ContainerStarted","Data":"6715c06e0f48b98e8a3b30617824b405f9d5954b9758518e62b43f03cd725f55"}
Oct 31 18:51:22.347680 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:22.347640    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r" event={"ID":"bd67e094-084a-4c09-bc37-1ec4a40cc2da","Type":"ContainerStarted","Data":"6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589"}
Oct 31 18:51:22.348070 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:22.347734    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:22.464181 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:22.464131    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r" podStartSLOduration=2.63054672 podStartE2EDuration="3.464118257s" totalImagesPullingTime="833.571537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:51:22.461316191 +0000 UTC m=+1828.607766053" watchObservedRunningTime="2026-10-31 18:51:22.464118257 +0000 UTC m=+1828.610568126"
Oct 31 18:51:24.725498 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:24.725433    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:25.470377 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:51:25.470343    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:51:28.359868 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:28.359842    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:51:29.344239 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:29.344196    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:30.166224 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:30.166180    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:51:33.691791 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.691755    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"]
Oct 31 18:51:33.697101 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.697086    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:33.803281 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.803247    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-335e78-serving-cert\""
Oct 31 18:51:33.806500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.806466    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"]
Oct 31 18:51:33.880013 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.879974    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-metrics-certs\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:33.880013 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.880015    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:33.880223 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.880100    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g9hw\" (UniqueName: \"kubernetes.io/projected/ef8decda-dfc2-4801-89dc-8be430938bfa-kube-api-access-8g9hw\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:33.880223 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.880209    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8decda-dfc2-4801-89dc-8be430938bfa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:33.910349 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.910319    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:33.981644 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.981554    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:33.981644 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.981620    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9hw\" (UniqueName: \"kubernetes.io/projected/ef8decda-dfc2-4801-89dc-8be430938bfa-kube-api-access-8g9hw\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:33.982217 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:51:33.981945    4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-serving-cert: secret "maas-api-e2e-ipp-b-335e78-serving-cert" not found
Oct 31 18:51:33.982217 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:51:33.982094    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-maas-api-tls podName:ef8decda-dfc2-4801-89dc-8be430938bfa nodeName:}" failed. No retries permitted until 2026-10-31 18:51:34.482069935 +0000 UTC m=+1840.628519781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-maas-api-tls") pod "maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k" (UID: "ef8decda-dfc2-4801-89dc-8be430938bfa") : secret "maas-api-e2e-ipp-b-335e78-serving-cert" not found
Oct 31 18:51:33.983507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.983447    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8decda-dfc2-4801-89dc-8be430938bfa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:33.983646 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.983633    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-metrics-certs\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:33.984042 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.984025    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8decda-dfc2-4801-89dc-8be430938bfa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:33.986597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:33.986571    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-metrics-certs\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:34.041770 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:34.041737    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g9hw\" (UniqueName: \"kubernetes.io/projected/ef8decda-dfc2-4801-89dc-8be430938bfa-kube-api-access-8g9hw\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:34.067916 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:34.067885    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:34.219572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:34.219536    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:34.491092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:34.491060    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:34.493619 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:34.493598    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:34.610571 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:34.610546    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:34.988238 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:34.988209    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"]
Oct 31 18:51:34.988691 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:51:34.988673    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8decda_dfc2_4801_89dc_8be430938bfa.slice/crio-67bf15a817b6b4b76ed154e65b672199f97345882f2c794c6535a3cf9f97ebe4 WatchSource:0}: Error finding container 67bf15a817b6b4b76ed154e65b672199f97345882f2c794c6535a3cf9f97ebe4: Status 404 returned error can't find the container with id 67bf15a817b6b4b76ed154e65b672199f97345882f2c794c6535a3cf9f97ebe4
Oct 31 18:51:35.433397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:35.433356    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k" event={"ID":"ef8decda-dfc2-4801-89dc-8be430938bfa","Type":"ContainerStarted","Data":"67bf15a817b6b4b76ed154e65b672199f97345882f2c794c6535a3cf9f97ebe4"}
Oct 31 18:51:36.442542 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:36.442506    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k" event={"ID":"ef8decda-dfc2-4801-89dc-8be430938bfa","Type":"ContainerStarted","Data":"addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729"}
Oct 31 18:51:36.442962 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:36.442573    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:36.557366 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:36.557302    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k" podStartSLOduration=2.722705253 podStartE2EDuration="3.557288642s" totalImagesPullingTime="834.583389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:51:36.557135032 +0000 UTC m=+1842.703584909" watchObservedRunningTime="2026-10-31 18:51:36.557288642 +0000 UTC m=+1842.703738507"
Oct 31 18:51:38.918635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:38.918594    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:42.408741 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:42.408704    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:42.458211 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:42.458178    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:51:42.893719 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:42.893687    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:44.784924 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:44.784889    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"]
Oct 31 18:51:44.789184 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:44.789165    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"
Oct 31 18:51:44.820867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:44.820834    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxw6j\" (UniqueName: \"kubernetes.io/projected/711334f9-d214-4503-bf64-4e0b8bc408a6-kube-api-access-pxw6j\") pod \"payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q\" (UID: \"711334f9-d214-4503-bf64-4e0b8bc408a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"
Oct 31 18:51:44.821052 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:44.820973    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711334f9-d214-4503-bf64-4e0b8bc408a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q\" (UID: \"711334f9-d214-4503-bf64-4e0b8bc408a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"
Oct 31 18:51:44.844446 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:44.844415    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"]
Oct 31 18:51:44.891041 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:44.891003    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2a4923\""
Oct 31 18:51:44.891200 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:44.891107    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2a4923-dockercfg-rn6hj\""
Oct 31 18:51:44.922737 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:44.922699    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711334f9-d214-4503-bf64-4e0b8bc408a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q\" (UID: \"711334f9-d214-4503-bf64-4e0b8bc408a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"
Oct 31 18:51:44.922901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:44.922850    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxw6j\" (UniqueName: \"kubernetes.io/projected/711334f9-d214-4503-bf64-4e0b8bc408a6-kube-api-access-pxw6j\") pod \"payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q\" (UID: \"711334f9-d214-4503-bf64-4e0b8bc408a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"
Oct 31 18:51:44.923377 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:44.923358    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711334f9-d214-4503-bf64-4e0b8bc408a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q\" (UID: \"711334f9-d214-4503-bf64-4e0b8bc408a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"
Oct 31 18:51:44.980541 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:44.980512    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxw6j\" (UniqueName: \"kubernetes.io/projected/711334f9-d214-4503-bf64-4e0b8bc408a6-kube-api-access-pxw6j\") pod \"payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q\" (UID: \"711334f9-d214-4503-bf64-4e0b8bc408a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"
Oct 31 18:51:45.103403 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:45.103319    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"
Oct 31 18:51:45.402856 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:45.402779    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:45.481702 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:45.481671    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"]
Oct 31 18:51:45.481880 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:51:45.481783    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711334f9_d214_4503_bf64_4e0b8bc408a6.slice/crio-1224993ef8cc43884dfc7ed6295cd9580b101366e89d9f409a05cdc08cd82912 WatchSource:0}: Error finding container 1224993ef8cc43884dfc7ed6295cd9580b101366e89d9f409a05cdc08cd82912: Status 404 returned error can't find the container with id 1224993ef8cc43884dfc7ed6295cd9580b101366e89d9f409a05cdc08cd82912
Oct 31 18:51:45.511229 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:45.511203    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q" event={"ID":"711334f9-d214-4503-bf64-4e0b8bc408a6","Type":"ContainerStarted","Data":"1224993ef8cc43884dfc7ed6295cd9580b101366e89d9f409a05cdc08cd82912"}
Oct 31 18:51:47.528126 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:47.528091    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q" event={"ID":"711334f9-d214-4503-bf64-4e0b8bc408a6","Type":"ContainerStarted","Data":"c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88"}
Oct 31 18:51:47.528643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:47.528150    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"
Oct 31 18:51:47.645257 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:47.645206    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q" podStartSLOduration=1.92203707 podStartE2EDuration="3.645192649s" totalImagesPullingTime="1.723155579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:51:47.642406068 +0000 UTC m=+1853.788855934" watchObservedRunningTime="2026-10-31 18:51:47.645192649 +0000 UTC m=+1853.791642513"
Oct 31 18:51:50.136493 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:50.136432    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:50.341540 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:50.341504    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:50.495384 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:50.495357    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:50.677882 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:50.677840    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:50.845330 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:50.845239    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:54.686778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:54.686747    4484 scope.go:122] "RemoveContainer" containerID="040161d0211a2dbe56dbd0008bfe36617331ad9c31ecddc120f7083686fdb94a"
Oct 31 18:51:54.850761 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:54.850725    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:55.126319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:55.126267    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:51:55.468361 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:51:55.468278    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:51:58.536293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:58.536264    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"
Oct 31 18:51:59.735560 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:51:59.735521    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:05.893684 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:05.893652    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:06.047536 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:06.047455    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:06.190555 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:06.190466    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:06.651731 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:06.651699    4484 generic.go:356] "Generic (PLEG): container finished" podID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerID="1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4" exitCode=0
Oct 31 18:52:06.651912 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:06.651743    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" event={"ID":"c5d585fa-0b3c-4049-a004-c5fab33a5f2c","Type":"ContainerDied","Data":"1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4"}
Oct 31 18:52:06.652124 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:06.652109    4484 scope.go:122] "RemoveContainer" containerID="1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4"
Oct 31 18:52:06.653721 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:06.653688    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:52:06.653819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:06.653767    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:52:06.653819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:06.653801    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 18:52:07.659792 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:07.659763    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" event={"ID":"c5d585fa-0b3c-4049-a004-c5fab33a5f2c","Type":"ContainerStarted","Data":"7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5"}
Oct 31 18:52:08.324530 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:08.324499    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:52:08.324719 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:08.324640    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 18:52:08.324719 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:08.324683    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 18:52:09.324565 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:09.324529    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 18:52:09.324969 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:09.324580    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 18:52:10.324588 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:10.324560    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 18:52:10.325016 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:10.324606    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 18:52:10.890186 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:10.890148    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:11.325148 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:11.325115    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 18:52:11.325567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:11.325184    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 18:52:12.324744 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:12.324714    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 18:52:12.324926 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:12.324772    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 18:52:13.329512 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:13.329460    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:52:13.329880 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:13.329863    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:52:13.330566 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:13.330546    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:52:14.342424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:14.342391    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:52:17.147235 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:17.147200    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:21.984401 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:21.984366    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:24.933825 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:24.933789    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:25.470663 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:52:25.470626    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:52:25.757597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:25.757550    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:26.830624 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:26.830589    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:27.083650 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:27.083564    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:27.361704 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:27.361617    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:31.156840 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:31.156803    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:31.344216 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:31.344181    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:31.518309 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:31.518276    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:32.684055 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:32.684019    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:34.348271 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.348238    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:34.546735 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.546701    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:34.735897 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.735826    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"]
Oct 31 18:52:34.740069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.740017    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"
Oct 31 18:52:34.798686 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.798659    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"]
Oct 31 18:52:34.845679 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.845648    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e9c518\""
Oct 31 18:52:34.863256 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.852603    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e9c518-dockercfg-7dnpj\""
Oct 31 18:52:34.870372 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.863608    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvq6l\" (UniqueName: \"kubernetes.io/projected/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-kube-api-access-cvq6l\") pod \"payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps\" (UID: \"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"
Oct 31 18:52:34.870372 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.863852    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps\" (UID: \"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"
Oct 31 18:52:34.894643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.894613    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"]
Oct 31 18:52:34.898737 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.898716    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"
Oct 31 18:52:34.954239 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.954207    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"]
Oct 31 18:52:34.964996 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.964965    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79249a1b-9ab2-4969-95b5-51777c695a77-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc\" (UID: \"79249a1b-9ab2-4969-95b5-51777c695a77\") " pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"
Oct 31 18:52:34.965170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.965114    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvq6l\" (UniqueName: \"kubernetes.io/projected/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-kube-api-access-cvq6l\") pod \"payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps\" (UID: \"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"
Oct 31 18:52:34.965245 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.965214    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps\" (UID: \"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"
Oct 31 18:52:34.965313 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.965243    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjk4q\" (UniqueName: \"kubernetes.io/projected/79249a1b-9ab2-4969-95b5-51777c695a77-kube-api-access-gjk4q\") pod \"payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc\" (UID: \"79249a1b-9ab2-4969-95b5-51777c695a77\") " pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"
Oct 31 18:52:34.965959 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:34.965938    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps\" (UID: \"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"
Oct 31 18:52:35.022819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.022793    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvq6l\" (UniqueName: \"kubernetes.io/projected/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-kube-api-access-cvq6l\") pod \"payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps\" (UID: \"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"
Oct 31 18:52:35.052964 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.052941    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"
Oct 31 18:52:35.066788 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.066759    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79249a1b-9ab2-4969-95b5-51777c695a77-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc\" (UID: \"79249a1b-9ab2-4969-95b5-51777c695a77\") " pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"
Oct 31 18:52:35.067082 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.067062    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjk4q\" (UniqueName: \"kubernetes.io/projected/79249a1b-9ab2-4969-95b5-51777c695a77-kube-api-access-gjk4q\") pod \"payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc\" (UID: \"79249a1b-9ab2-4969-95b5-51777c695a77\") " pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"
Oct 31 18:52:35.067604 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.067582    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79249a1b-9ab2-4969-95b5-51777c695a77-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc\" (UID: \"79249a1b-9ab2-4969-95b5-51777c695a77\") " pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"
Oct 31 18:52:35.133053 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.128031    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjk4q\" (UniqueName: \"kubernetes.io/projected/79249a1b-9ab2-4969-95b5-51777c695a77-kube-api-access-gjk4q\") pod \"payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc\" (UID: \"79249a1b-9ab2-4969-95b5-51777c695a77\") " pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"
Oct 31 18:52:35.214223 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.214188    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"
Oct 31 18:52:35.424497 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.424454    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"]
Oct 31 18:52:35.424897 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:52:35.424875    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3f0f92_f8e1_41f2_a5e6_5911bfff0c71.slice/crio-35131ca1e423325babfb13c07d10c391499b8f9257225578548ac3554d7ec90e WatchSource:0}: Error finding container 35131ca1e423325babfb13c07d10c391499b8f9257225578548ac3554d7ec90e: Status 404 returned error can't find the container with id 35131ca1e423325babfb13c07d10c391499b8f9257225578548ac3554d7ec90e
Oct 31 18:52:35.588695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.588671    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"]
Oct 31 18:52:35.589093 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:52:35.589072    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79249a1b_9ab2_4969_95b5_51777c695a77.slice/crio-608a846c9c81847417f30fbd668e534f63628f6f0f746ce5305cc2b37e28f7d0 WatchSource:0}: Error finding container 608a846c9c81847417f30fbd668e534f63628f6f0f746ce5305cc2b37e28f7d0: Status 404 returned error can't find the container with id 608a846c9c81847417f30fbd668e534f63628f6f0f746ce5305cc2b37e28f7d0
Oct 31 18:52:35.878885 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.878797    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps" event={"ID":"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71","Type":"ContainerStarted","Data":"8c2883b56f525874716366b23edc5916ff52f287a3b9587fd623a76c3620d2a4"}
Oct 31 18:52:35.878885 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.878855    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps" event={"ID":"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71","Type":"ContainerStarted","Data":"35131ca1e423325babfb13c07d10c391499b8f9257225578548ac3554d7ec90e"}
Oct 31 18:52:35.878885 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.878884    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"
Oct 31 18:52:35.880401 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.880378    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc" event={"ID":"79249a1b-9ab2-4969-95b5-51777c695a77","Type":"ContainerStarted","Data":"d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f"}
Oct 31 18:52:35.880505 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.880424    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc" event={"ID":"79249a1b-9ab2-4969-95b5-51777c695a77","Type":"ContainerStarted","Data":"608a846c9c81847417f30fbd668e534f63628f6f0f746ce5305cc2b37e28f7d0"}
Oct 31 18:52:35.880505 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.880452    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"
Oct 31 18:52:35.993205 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:35.993160    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps" podStartSLOduration=1.993148138 podStartE2EDuration="1.993148138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:52:35.991338966 +0000 UTC m=+1902.137788829" watchObservedRunningTime="2026-10-31 18:52:35.993148138 +0000 UTC m=+1902.139598001"
Oct 31 18:52:36.106938 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:36.106884    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc" podStartSLOduration=2.106869683 podStartE2EDuration="2.106869683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:52:36.105517867 +0000 UTC m=+1902.251967734" watchObservedRunningTime="2026-10-31 18:52:36.106869683 +0000 UTC m=+1902.253319547"
Oct 31 18:52:38.094746 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:38.094711    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:40.335018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:40.334990    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"]
Oct 31 18:52:40.335418 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:40.335191    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" podUID="a4415ab7-0e82-435f-b701-544af2b06359" containerName="istio-proxy" containerID="cri-o://c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5" gracePeriod=30
Oct 31 18:52:41.566243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:41.566212    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:42.633675 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:42.633641    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:42.783599 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:42.783565    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:43.272091 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:43.272056    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:43.421028 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:43.420984    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:44.452273 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:44.452232    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:52:44.452677 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:44.452291    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" podUID="a4415ab7-0e82-435f-b701-544af2b06359" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:52:45.613652 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.613587    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:52:45.691199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691171    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-socket\") pod \"a4415ab7-0e82-435f-b701-544af2b06359\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") "
Oct 31 18:52:45.691347 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691216    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-kube-api-access-sw9x4\" (UniqueName: \"kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-kube-api-access-sw9x4\") pod \"a4415ab7-0e82-435f-b701-544af2b06359\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") "
Oct 31 18:52:45.691347 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691245    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-credential-socket\") pod \"a4415ab7-0e82-435f-b701-544af2b06359\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") "
Oct 31 18:52:45.691347 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691279    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-data\") pod \"a4415ab7-0e82-435f-b701-544af2b06359\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") "
Oct 31 18:52:45.691347 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691327    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4415ab7-0e82-435f-b701-544af2b06359-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4415ab7-0e82-435f-b701-544af2b06359-istio-podinfo\") pod \"a4415ab7-0e82-435f-b701-544af2b06359\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") "
Oct 31 18:52:45.691575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691388    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-certs\") pod \"a4415ab7-0e82-435f-b701-544af2b06359\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") "
Oct 31 18:52:45.691575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691432    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4415ab7-0e82-435f-b701-544af2b06359-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4415ab7-0e82-435f-b701-544af2b06359-istiod-ca-cert\") pod \"a4415ab7-0e82-435f-b701-544af2b06359\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") "
Oct 31 18:52:45.691575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691450    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-socket" pod "a4415ab7-0e82-435f-b701-544af2b06359" (UID: "a4415ab7-0e82-435f-b701-544af2b06359"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:52:45.691575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691520    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-istio-token\" (UniqueName: \"kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-istio-token\") pod \"a4415ab7-0e82-435f-b701-544af2b06359\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") "
Oct 31 18:52:45.691575 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691571    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-envoy\") pod \"a4415ab7-0e82-435f-b701-544af2b06359\" (UID: \"a4415ab7-0e82-435f-b701-544af2b06359\") "
Oct 31 18:52:45.691819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691613    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-credential-socket" pod "a4415ab7-0e82-435f-b701-544af2b06359" (UID: "a4415ab7-0e82-435f-b701-544af2b06359"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:52:45.691819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691657    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-certs" pod "a4415ab7-0e82-435f-b701-544af2b06359" (UID: "a4415ab7-0e82-435f-b701-544af2b06359"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:52:45.691819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691675    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-data" pod "a4415ab7-0e82-435f-b701-544af2b06359" (UID: "a4415ab7-0e82-435f-b701-544af2b06359"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:52:45.691966 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.691861    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4415ab7-0e82-435f-b701-544af2b06359-istiod-ca-cert" pod "a4415ab7-0e82-435f-b701-544af2b06359" (UID: "a4415ab7-0e82-435f-b701-544af2b06359"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:52:45.692147 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.692124    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:45.692275 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.692152    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:45.692275 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.692166    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:45.692275 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.692179    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:45.692275 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.692192    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4415ab7-0e82-435f-b701-544af2b06359-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:45.693561 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.693538    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-kube-api-access-sw9x4" pod "a4415ab7-0e82-435f-b701-544af2b06359" (UID: "a4415ab7-0e82-435f-b701-544af2b06359"). InnerVolumeSpecName "kube-api-access-sw9x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:52:45.693786 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.693768    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-istio-token" pod "a4415ab7-0e82-435f-b701-544af2b06359" (UID: "a4415ab7-0e82-435f-b701-544af2b06359"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:52:45.694121 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.694106    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4415ab7-0e82-435f-b701-544af2b06359-istio-podinfo" pod "a4415ab7-0e82-435f-b701-544af2b06359" (UID: "a4415ab7-0e82-435f-b701-544af2b06359"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:52:45.694178 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.694123    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-envoy" pod "a4415ab7-0e82-435f-b701-544af2b06359" (UID: "a4415ab7-0e82-435f-b701-544af2b06359"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:52:45.794208 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.794173    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:45.794341 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.794213    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4415ab7-0e82-435f-b701-544af2b06359-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:45.794341 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.794229    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw9x4\" (UniqueName: \"kubernetes.io/projected/a4415ab7-0e82-435f-b701-544af2b06359-kube-api-access-sw9x4\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:45.794341 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.794244    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4415ab7-0e82-435f-b701-544af2b06359-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:52:45.953865 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.953772    4484 generic.go:356] "Generic (PLEG): container finished" podID="a4415ab7-0e82-435f-b701-544af2b06359" containerID="c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5" exitCode=0
Oct 31 18:52:45.953865 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.953823    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"
Oct 31 18:52:45.954067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.953855    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" event={"ID":"a4415ab7-0e82-435f-b701-544af2b06359","Type":"ContainerDied","Data":"c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5"}
Oct 31 18:52:45.954067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.953915    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf" event={"ID":"a4415ab7-0e82-435f-b701-544af2b06359","Type":"ContainerDied","Data":"9b0673d28ac4709199d6903e354a44c4c98c94c72acb231ec84f306eb7ce9080"}
Oct 31 18:52:45.954067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.953936    4484 scope.go:122] "RemoveContainer" containerID="c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5"
Oct 31 18:52:45.964648 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.964632    4484 scope.go:122] "RemoveContainer" containerID="c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5"
Oct 31 18:52:45.964909 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:52:45.964888    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5\": container with ID starting with c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5 not found: ID does not exist" containerID="c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5"
Oct 31 18:52:45.964974 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:45.964922    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5"} err="failed to get container status \"c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5\": rpc error: code = NotFound desc = could not find container \"c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5\": container with ID starting with c7b3f2dcec756da3ae125decfe968535e860fc140c8505605084235f1ecc20a5 not found: ID does not exist"
Oct 31 18:52:46.120093 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:46.120059    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"]
Oct 31 18:52:46.124218 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:46.124197    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-d2qcf"]
Oct 31 18:52:46.891548 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:46.891519    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"
Oct 31 18:52:46.891957 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:46.891589    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"
Oct 31 18:52:47.253529 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:47.253499    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:52:47.653496 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:47.653401    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:47.993851 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:47.993819    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4415ab7-0e82-435f-b701-544af2b06359" path="/var/lib/kubelet/pods/a4415ab7-0e82-435f-b701-544af2b06359/volumes"
Oct 31 18:52:48.414789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:48.414715    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:49.403918 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:49.403881    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:51.020080 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:51.020043    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:53.712134 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:53.712096    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:55.225567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:55.225535    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"]
Oct 31 18:52:55.230503 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:55.230453    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:55.454466 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:55.454438    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:55.470074 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:52:55.470043    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:52:55.614769 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:55.614736    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:52:56.761606 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:56.761565    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"]
Oct 31 18:52:56.762112 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:56.761860    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" podUID="76199531-8025-41d1-813a-cc1a78125e06" containerName="istio-proxy" containerID="cri-o://925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723" gracePeriod=30
Oct 31 18:52:59.709739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:59.709707    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-hz57f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:52:59.710131 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:52:59.709760    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" podUID="76199531-8025-41d1-813a-cc1a78125e06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:53:00.226863 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:00.226826    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q" podUID="dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b" containerName="payload-pre-processing" containerID="cri-o://95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f" gracePeriod=25
Oct 31 18:53:00.508451 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:00.508413    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:00.518088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:00.518064    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"
Oct 31 18:53:00.672060 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:00.672029    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-kube-api-access-l95v7\" (UniqueName: \"kubernetes.io/projected/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-kube-api-access-l95v7\") pod \"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b\" (UID: \"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b\") "
Oct 31 18:53:00.672230 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:00.672070    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-plugins-config-volume\") pod \"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b\" (UID: \"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b\") "
Oct 31 18:53:00.672614 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:00.672587    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-plugins-config-volume" pod "dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b" (UID: "dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:53:00.674548 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:00.674521    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-kube-api-access-l95v7" pod "dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b" (UID: "dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b"). InnerVolumeSpecName "kube-api-access-l95v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:00.773341 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:00.773314    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l95v7\" (UniqueName: \"kubernetes.io/projected/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-kube-api-access-l95v7\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:00.773718 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:00.773344    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:01.048625 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:01.048546    4484 generic.go:356] "Generic (PLEG): container finished" podID="dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b" containerID="95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f" exitCode=0
Oct 31 18:53:01.048625 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:01.048595    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"
Oct 31 18:53:01.048802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:01.048632    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q" event={"ID":"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b","Type":"ContainerDied","Data":"95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f"}
Oct 31 18:53:01.048802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:01.048684    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q" event={"ID":"dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b","Type":"ContainerDied","Data":"3b389949093fb0f44fe74d12663b2bbbdf3d9221048b8365f9bfeb75636ba840"}
Oct 31 18:53:01.048802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:01.048706    4484 scope.go:122] "RemoveContainer" containerID="95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f"
Oct 31 18:53:01.058694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:01.058680    4484 scope.go:122] "RemoveContainer" containerID="95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f"
Oct 31 18:53:01.058962 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:53:01.058942    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f\": container with ID starting with 95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f not found: ID does not exist" containerID="95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f"
Oct 31 18:53:01.059006 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:01.058971    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f"} err="failed to get container status \"95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f\": rpc error: code = NotFound desc = could not find container \"95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f\": container with ID starting with 95258c1b57a181c92d9ac5564ede2ab52c1230f3fec6a3d2130e14f06e01c11f not found: ID does not exist"
Oct 31 18:53:01.214639 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:01.214613    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"]
Oct 31 18:53:01.264928 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:01.264898    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6b8cb48d7-s926q"]
Oct 31 18:53:01.995321 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:01.995290    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b" path="/var/lib/kubelet/pods/dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b/volumes"
Oct 31 18:53:02.026646 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.026624    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:53:02.064799 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.064714    4484 generic.go:356] "Generic (PLEG): container finished" podID="76199531-8025-41d1-813a-cc1a78125e06" containerID="925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723" exitCode=0
Oct 31 18:53:02.064799 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.064771    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"
Oct 31 18:53:02.064971 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.064802    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" event={"ID":"76199531-8025-41d1-813a-cc1a78125e06","Type":"ContainerDied","Data":"925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723"}
Oct 31 18:53:02.064971 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.064883    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f" event={"ID":"76199531-8025-41d1-813a-cc1a78125e06","Type":"ContainerDied","Data":"a329bd3a062d03c68a4570acad848285fbf42cc04d0b92285f335f1d102deb1d"}
Oct 31 18:53:02.064971 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.064906    4484 scope.go:122] "RemoveContainer" containerID="925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723"
Oct 31 18:53:02.074292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.074274    4484 scope.go:122] "RemoveContainer" containerID="925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723"
Oct 31 18:53:02.074558 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:53:02.074539    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723\": container with ID starting with 925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723 not found: ID does not exist" containerID="925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723"
Oct 31 18:53:02.074630 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.074569    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723"} err="failed to get container status \"925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723\": rpc error: code = NotFound desc = could not find container \"925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723\": container with ID starting with 925f8bb7eccb5f0925ec675e48f882f0ed2131078df52c81868ca8543bf0a723 not found: ID does not exist"
Oct 31 18:53:02.190708 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.190679    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-credential-socket\") pod \"76199531-8025-41d1-813a-cc1a78125e06\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") "
Oct 31 18:53:02.190867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.190736    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-data\") pod \"76199531-8025-41d1-813a-cc1a78125e06\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") "
Oct 31 18:53:02.190867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.190772    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-socket\") pod \"76199531-8025-41d1-813a-cc1a78125e06\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") "
Oct 31 18:53:02.190867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.190847    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-kube-api-access-bpdl6\" (UniqueName: \"kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-kube-api-access-bpdl6\") pod \"76199531-8025-41d1-813a-cc1a78125e06\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") "
Oct 31 18:53:02.190985 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.190883    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-certs\") pod \"76199531-8025-41d1-813a-cc1a78125e06\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") "
Oct 31 18:53:02.190985 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.190941    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-envoy\") pod \"76199531-8025-41d1-813a-cc1a78125e06\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") "
Oct 31 18:53:02.191088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191001    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-credential-socket" pod "76199531-8025-41d1-813a-cc1a78125e06" (UID: "76199531-8025-41d1-813a-cc1a78125e06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:02.191088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191026    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-istio-token\" (UniqueName: \"kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-istio-token\") pod \"76199531-8025-41d1-813a-cc1a78125e06\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") "
Oct 31 18:53:02.191088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191017    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-data" pod "76199531-8025-41d1-813a-cc1a78125e06" (UID: "76199531-8025-41d1-813a-cc1a78125e06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:02.191088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191049    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-socket" pod "76199531-8025-41d1-813a-cc1a78125e06" (UID: "76199531-8025-41d1-813a-cc1a78125e06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:02.191088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191085    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76199531-8025-41d1-813a-cc1a78125e06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76199531-8025-41d1-813a-cc1a78125e06-istiod-ca-cert\") pod \"76199531-8025-41d1-813a-cc1a78125e06\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") "
Oct 31 18:53:02.191347 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191172    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76199531-8025-41d1-813a-cc1a78125e06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76199531-8025-41d1-813a-cc1a78125e06-istio-podinfo\") pod \"76199531-8025-41d1-813a-cc1a78125e06\" (UID: \"76199531-8025-41d1-813a-cc1a78125e06\") "
Oct 31 18:53:02.191347 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191194    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-certs" pod "76199531-8025-41d1-813a-cc1a78125e06" (UID: "76199531-8025-41d1-813a-cc1a78125e06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:02.191546 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191522    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76199531-8025-41d1-813a-cc1a78125e06-istiod-ca-cert" pod "76199531-8025-41d1-813a-cc1a78125e06" (UID: "76199531-8025-41d1-813a-cc1a78125e06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:53:02.191781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191647    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:02.191781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191667    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:02.191781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191682    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:02.191781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191695    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:02.191781 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.191708    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76199531-8025-41d1-813a-cc1a78125e06-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:02.194444 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.194387    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76199531-8025-41d1-813a-cc1a78125e06-istio-podinfo" pod "76199531-8025-41d1-813a-cc1a78125e06" (UID: "76199531-8025-41d1-813a-cc1a78125e06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:53:02.195111 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.195089    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-envoy" pod "76199531-8025-41d1-813a-cc1a78125e06" (UID: "76199531-8025-41d1-813a-cc1a78125e06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:53:02.195886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.195859    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-kube-api-access-bpdl6" pod "76199531-8025-41d1-813a-cc1a78125e06" (UID: "76199531-8025-41d1-813a-cc1a78125e06"). InnerVolumeSpecName "kube-api-access-bpdl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:02.196859 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.196822    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-istio-token" pod "76199531-8025-41d1-813a-cc1a78125e06" (UID: "76199531-8025-41d1-813a-cc1a78125e06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:02.292530 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.292488    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:02.292530 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.292518    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76199531-8025-41d1-813a-cc1a78125e06-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:02.292530 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.292530    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpdl6\" (UniqueName: \"kubernetes.io/projected/76199531-8025-41d1-813a-cc1a78125e06-kube-api-access-bpdl6\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:02.292530 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.292544    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76199531-8025-41d1-813a-cc1a78125e06-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:02.487295 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.487262    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"]
Oct 31 18:53:02.489919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:02.489897    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-hz57f"]
Oct 31 18:53:03.994149 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:03.994118    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76199531-8025-41d1-813a-cc1a78125e06" path="/var/lib/kubelet/pods/76199531-8025-41d1-813a-cc1a78125e06/volumes"
Oct 31 18:53:05.344296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:05.344256    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:10.058915 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:10.058871    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:15.399248 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:15.399213    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:15.641790 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:15.641759    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"]
Oct 31 18:53:15.749407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:15.749374    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"]
Oct 31 18:53:15.893537 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:15.893503    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:16.050707 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:16.050620    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:16.214202 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:16.214171    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:20.642351 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:20.642306    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch" podUID="46cc80b3-2af6-442e-9080-8576b60ee024" containerName="payload-processing" containerID="cri-o://32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7" gracePeriod=25
Oct 31 18:53:20.750236 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:20.750197    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v" podUID="416036a1-61e2-4b66-92f7-3236cedbfcd4" containerName="payload-pre-processing" containerID="cri-o://dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e" gracePeriod=25
Oct 31 18:53:20.911032 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:20.910967    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"
Oct 31 18:53:20.999710 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:20.999684    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46cc80b3-2af6-442e-9080-8576b60ee024-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46cc80b3-2af6-442e-9080-8576b60ee024-plugins-config-volume\") pod \"46cc80b3-2af6-442e-9080-8576b60ee024\" (UID: \"46cc80b3-2af6-442e-9080-8576b60ee024\") "
Oct 31 18:53:20.999892 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:20.999786    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cc80b3-2af6-442e-9080-8576b60ee024-kube-api-access-pd6p7\" (UniqueName: \"kubernetes.io/projected/46cc80b3-2af6-442e-9080-8576b60ee024-kube-api-access-pd6p7\") pod \"46cc80b3-2af6-442e-9080-8576b60ee024\" (UID: \"46cc80b3-2af6-442e-9080-8576b60ee024\") "
Oct 31 18:53:21.000075 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.000052    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cc80b3-2af6-442e-9080-8576b60ee024-plugins-config-volume" pod "46cc80b3-2af6-442e-9080-8576b60ee024" (UID: "46cc80b3-2af6-442e-9080-8576b60ee024"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:53:21.000244 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.000227    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46cc80b3-2af6-442e-9080-8576b60ee024-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:21.002041 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.002018    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cc80b3-2af6-442e-9080-8576b60ee024-kube-api-access-pd6p7" pod "46cc80b3-2af6-442e-9080-8576b60ee024" (UID: "46cc80b3-2af6-442e-9080-8576b60ee024"). InnerVolumeSpecName "kube-api-access-pd6p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:21.020045 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.020023    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"
Oct 31 18:53:21.101162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.101133    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416036a1-61e2-4b66-92f7-3236cedbfcd4-kube-api-access-htnww\" (UniqueName: \"kubernetes.io/projected/416036a1-61e2-4b66-92f7-3236cedbfcd4-kube-api-access-htnww\") pod \"416036a1-61e2-4b66-92f7-3236cedbfcd4\" (UID: \"416036a1-61e2-4b66-92f7-3236cedbfcd4\") "
Oct 31 18:53:21.101348 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.101335    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416036a1-61e2-4b66-92f7-3236cedbfcd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416036a1-61e2-4b66-92f7-3236cedbfcd4-plugins-config-volume\") pod \"416036a1-61e2-4b66-92f7-3236cedbfcd4\" (UID: \"416036a1-61e2-4b66-92f7-3236cedbfcd4\") "
Oct 31 18:53:21.101757 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.101726    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416036a1-61e2-4b66-92f7-3236cedbfcd4-plugins-config-volume" pod "416036a1-61e2-4b66-92f7-3236cedbfcd4" (UID: "416036a1-61e2-4b66-92f7-3236cedbfcd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:53:21.101891 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.101831    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd6p7\" (UniqueName: \"kubernetes.io/projected/46cc80b3-2af6-442e-9080-8576b60ee024-kube-api-access-pd6p7\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:21.103363 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.103345    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416036a1-61e2-4b66-92f7-3236cedbfcd4-kube-api-access-htnww" pod "416036a1-61e2-4b66-92f7-3236cedbfcd4" (UID: "416036a1-61e2-4b66-92f7-3236cedbfcd4"). InnerVolumeSpecName "kube-api-access-htnww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:21.192369 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.192281    4484 generic.go:356] "Generic (PLEG): container finished" podID="416036a1-61e2-4b66-92f7-3236cedbfcd4" containerID="dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e" exitCode=0
Oct 31 18:53:21.192369 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.192338    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"
Oct 31 18:53:21.192369 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.192347    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v" event={"ID":"416036a1-61e2-4b66-92f7-3236cedbfcd4","Type":"ContainerDied","Data":"dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e"}
Oct 31 18:53:21.192687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.192414    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v" event={"ID":"416036a1-61e2-4b66-92f7-3236cedbfcd4","Type":"ContainerDied","Data":"5d9b65d3c4886e35c36f8c8722742c6e361dabd8743a056320b7fa3398745dd2"}
Oct 31 18:53:21.192687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.192444    4484 scope.go:122] "RemoveContainer" containerID="dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e"
Oct 31 18:53:21.193976 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.193956    4484 generic.go:356] "Generic (PLEG): container finished" podID="46cc80b3-2af6-442e-9080-8576b60ee024" containerID="32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7" exitCode=0
Oct 31 18:53:21.194057 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.194029    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"
Oct 31 18:53:21.194105 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.194057    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch" event={"ID":"46cc80b3-2af6-442e-9080-8576b60ee024","Type":"ContainerDied","Data":"32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7"}
Oct 31 18:53:21.194105 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.194096    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch" event={"ID":"46cc80b3-2af6-442e-9080-8576b60ee024","Type":"ContainerDied","Data":"46ec049b5a72567f19fb28cee6239cd8948fe372d0efeccaf1466d9b5cdfce26"}
Oct 31 18:53:21.202857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.202831    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/416036a1-61e2-4b66-92f7-3236cedbfcd4-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:21.202965 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.202861    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htnww\" (UniqueName: \"kubernetes.io/projected/416036a1-61e2-4b66-92f7-3236cedbfcd4-kube-api-access-htnww\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:21.203198 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.203181    4484 scope.go:122] "RemoveContainer" containerID="dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e"
Oct 31 18:53:21.203562 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:53:21.203456    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e\": container with ID starting with dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e not found: ID does not exist" containerID="dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e"
Oct 31 18:53:21.203562 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.203502    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e"} err="failed to get container status \"dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e\": rpc error: code = NotFound desc = could not find container \"dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e\": container with ID starting with dadb2ff6bdaa27db889d14c69691c01d973b5765c43b3a5e5d0fbd5df841e66e not found: ID does not exist"
Oct 31 18:53:21.203562 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.203524    4484 scope.go:122] "RemoveContainer" containerID="32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7"
Oct 31 18:53:21.212797 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.212783    4484 scope.go:122] "RemoveContainer" containerID="32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7"
Oct 31 18:53:21.213034 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:53:21.213018    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7\": container with ID starting with 32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7 not found: ID does not exist" containerID="32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7"
Oct 31 18:53:21.213095 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.213039    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7"} err="failed to get container status \"32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7\": rpc error: code = NotFound desc = could not find container \"32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7\": container with ID starting with 32123fe70b13d6891aabff94839869281031f06cc1e2bf3895fcd694ecb782c7 not found: ID does not exist"
Oct 31 18:53:21.357049 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.357021    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"]
Oct 31 18:53:21.362027 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.362006    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-86b74d4879-7qmch"]
Oct 31 18:53:21.470999 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.470941    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"]
Oct 31 18:53:21.476162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.476142    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-7f445d5f5f-f4d6v"]
Oct 31 18:53:21.923886 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.923859    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:21.994525 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.994468    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416036a1-61e2-4b66-92f7-3236cedbfcd4" path="/var/lib/kubelet/pods/416036a1-61e2-4b66-92f7-3236cedbfcd4/volumes"
Oct 31 18:53:21.994990 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:21.994970    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46cc80b3-2af6-442e-9080-8576b60ee024" path="/var/lib/kubelet/pods/46cc80b3-2af6-442e-9080-8576b60ee024/volumes"
Oct 31 18:53:25.468233 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:53:25.468194    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:53:25.994359 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:25.994314    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:26.512368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:26.512243    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:26.717264 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:26.717220    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:27.942862 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:27.942827    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:29.426827 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:29.426791    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:29.584829 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:29.584795    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:30.983747 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:30.983714    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:31.483944 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:31.483904    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:31.889949 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:31.889919    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"]
Oct 31 18:53:31.999339 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:31.999309    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"]
Oct 31 18:53:32.088461 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:32.088426    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:32.242011 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:32.241952    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:32.399741 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:32.399702    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:33.123096 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:33.123059    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:36.890448 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:36.890414    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv" podUID="ea5c4327-1827-43f0-8f1e-b711df9dacb6" containerName="payload-processing" containerID="cri-o://6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433" gracePeriod=25
Oct 31 18:53:36.999633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:36.999596    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7" podUID="f62830c2-01ae-4f6d-acb2-5ab602d17919" containerName="payload-pre-processing" containerID="cri-o://345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e" gracePeriod=25
Oct 31 18:53:37.034146 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.034118    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:37.179233 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.179208    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"
Oct 31 18:53:37.192883 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.192857    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5c4327-1827-43f0-8f1e-b711df9dacb6-kube-api-access-h98t4\" (UniqueName: \"kubernetes.io/projected/ea5c4327-1827-43f0-8f1e-b711df9dacb6-kube-api-access-h98t4\") pod \"ea5c4327-1827-43f0-8f1e-b711df9dacb6\" (UID: \"ea5c4327-1827-43f0-8f1e-b711df9dacb6\") "
Oct 31 18:53:37.193026 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.192933    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5c4327-1827-43f0-8f1e-b711df9dacb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c4327-1827-43f0-8f1e-b711df9dacb6-plugins-config-volume\") pod \"ea5c4327-1827-43f0-8f1e-b711df9dacb6\" (UID: \"ea5c4327-1827-43f0-8f1e-b711df9dacb6\") "
Oct 31 18:53:37.193269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.193250    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5c4327-1827-43f0-8f1e-b711df9dacb6-plugins-config-volume" pod "ea5c4327-1827-43f0-8f1e-b711df9dacb6" (UID: "ea5c4327-1827-43f0-8f1e-b711df9dacb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:53:37.193354 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.193331    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c4327-1827-43f0-8f1e-b711df9dacb6-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:37.195676 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.195653    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5c4327-1827-43f0-8f1e-b711df9dacb6-kube-api-access-h98t4" pod "ea5c4327-1827-43f0-8f1e-b711df9dacb6" (UID: "ea5c4327-1827-43f0-8f1e-b711df9dacb6"). InnerVolumeSpecName "kube-api-access-h98t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:37.276902 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.276877    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"
Oct 31 18:53:37.294322 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.294297    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f62830c2-01ae-4f6d-acb2-5ab602d17919-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62830c2-01ae-4f6d-acb2-5ab602d17919-plugins-config-volume\") pod \"f62830c2-01ae-4f6d-acb2-5ab602d17919\" (UID: \"f62830c2-01ae-4f6d-acb2-5ab602d17919\") "
Oct 31 18:53:37.294509 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.294466    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62830c2-01ae-4f6d-acb2-5ab602d17919-kube-api-access-956kb\" (UniqueName: \"kubernetes.io/projected/f62830c2-01ae-4f6d-acb2-5ab602d17919-kube-api-access-956kb\") pod \"f62830c2-01ae-4f6d-acb2-5ab602d17919\" (UID: \"f62830c2-01ae-4f6d-acb2-5ab602d17919\") "
Oct 31 18:53:37.294899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.294880    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h98t4\" (UniqueName: \"kubernetes.io/projected/ea5c4327-1827-43f0-8f1e-b711df9dacb6-kube-api-access-h98t4\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:37.295642 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.295609    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62830c2-01ae-4f6d-acb2-5ab602d17919-plugins-config-volume" pod "f62830c2-01ae-4f6d-acb2-5ab602d17919" (UID: "f62830c2-01ae-4f6d-acb2-5ab602d17919"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:53:37.296939 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.296919    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62830c2-01ae-4f6d-acb2-5ab602d17919-kube-api-access-956kb" pod "f62830c2-01ae-4f6d-acb2-5ab602d17919" (UID: "f62830c2-01ae-4f6d-acb2-5ab602d17919"). InnerVolumeSpecName "kube-api-access-956kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:37.311638 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.311609    4484 generic.go:356] "Generic (PLEG): container finished" podID="ea5c4327-1827-43f0-8f1e-b711df9dacb6" containerID="6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433" exitCode=0
Oct 31 18:53:37.311832 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.311818    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"
Oct 31 18:53:37.312554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.312296    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv" event={"ID":"ea5c4327-1827-43f0-8f1e-b711df9dacb6","Type":"ContainerDied","Data":"6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433"}
Oct 31 18:53:37.312554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.312373    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv" event={"ID":"ea5c4327-1827-43f0-8f1e-b711df9dacb6","Type":"ContainerDied","Data":"67d213b0c625b7ab359a625ce409e31ed4395cf377c6c1a2b978b2d4985e37df"}
Oct 31 18:53:37.312554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.312401    4484 scope.go:122] "RemoveContainer" containerID="6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433"
Oct 31 18:53:37.314339 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.314315    4484 generic.go:356] "Generic (PLEG): container finished" podID="f62830c2-01ae-4f6d-acb2-5ab602d17919" containerID="345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e" exitCode=0
Oct 31 18:53:37.314447 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.314438    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7" event={"ID":"f62830c2-01ae-4f6d-acb2-5ab602d17919","Type":"ContainerDied","Data":"345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e"}
Oct 31 18:53:37.314534 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.314501    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7" event={"ID":"f62830c2-01ae-4f6d-acb2-5ab602d17919","Type":"ContainerDied","Data":"6c6ad61b3d6cabb7f57629c6b3088d4415d185b66b7119fe0c58e692f814c0c1"}
Oct 31 18:53:37.314599 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.314552    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"
Oct 31 18:53:37.331897 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.331878    4484 scope.go:122] "RemoveContainer" containerID="6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433"
Oct 31 18:53:37.332237 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:53:37.332217    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433\": container with ID starting with 6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433 not found: ID does not exist" containerID="6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433"
Oct 31 18:53:37.332281 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.332244    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433"} err="failed to get container status \"6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433\": rpc error: code = NotFound desc = could not find container \"6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433\": container with ID starting with 6d02bdca8ebeb01b242f77bf82d491831fe291abc9488f493cf38b2f52111433 not found: ID does not exist"
Oct 31 18:53:37.332281 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.332261    4484 scope.go:122] "RemoveContainer" containerID="345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e"
Oct 31 18:53:37.341367 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.341336    4484 scope.go:122] "RemoveContainer" containerID="345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e"
Oct 31 18:53:37.341643 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:53:37.341622    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e\": container with ID starting with 345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e not found: ID does not exist" containerID="345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e"
Oct 31 18:53:37.341732 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.341653    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e"} err="failed to get container status \"345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e\": rpc error: code = NotFound desc = could not find container \"345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e\": container with ID starting with 345b32c7675cc7692dfedaf43f1ef79562c1680345ab4342d73ea7229de95e0e not found: ID does not exist"
Oct 31 18:53:37.395819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.395791    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-956kb\" (UniqueName: \"kubernetes.io/projected/f62830c2-01ae-4f6d-acb2-5ab602d17919-kube-api-access-956kb\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:37.395819 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.395813    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62830c2-01ae-4f6d-acb2-5ab602d17919-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:37.522645 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.522610    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"]
Oct 31 18:53:37.525230 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.525206    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-676224-cc6bc7cf5-5rzs7"]
Oct 31 18:53:37.633911 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.633877    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"]
Oct 31 18:53:37.639012 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.638981    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-676224-66bf75d57f-hvhjv"]
Oct 31 18:53:37.953174 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.953148    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:37.997018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.996989    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5c4327-1827-43f0-8f1e-b711df9dacb6" path="/var/lib/kubelet/pods/ea5c4327-1827-43f0-8f1e-b711df9dacb6/volumes"
Oct 31 18:53:37.997419 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:37.997405    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62830c2-01ae-4f6d-acb2-5ab602d17919" path="/var/lib/kubelet/pods/f62830c2-01ae-4f6d-acb2-5ab602d17919/volumes"
Oct 31 18:53:42.497791 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:42.497752    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:42.622055 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:42.622013    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:53:42.657011 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:42.656969    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:42.849682 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:42.849585    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:43.050359 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:43.050324    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:46.929528 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:46.929497    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"]
Oct 31 18:53:46.929980 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:46.929731    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r" podUID="bd67e094-084a-4c09-bc37-1ec4a40cc2da" containerName="maas-api" containerID="cri-o://6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589" gracePeriod=30
Oct 31 18:53:47.054647 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.054618    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:47.196918 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.196861    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:53:47.300214 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.300185    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd67e094-084a-4c09-bc37-1ec4a40cc2da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd67e094-084a-4c09-bc37-1ec4a40cc2da-openshift-service-ca\") pod \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") "
Oct 31 18:53:47.300417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.300290    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd67e094-084a-4c09-bc37-1ec4a40cc2da-kube-api-access-7ks4c\" (UniqueName: \"kubernetes.io/projected/bd67e094-084a-4c09-bc37-1ec4a40cc2da-kube-api-access-7ks4c\") pod \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") "
Oct 31 18:53:47.300417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.300353    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-metrics-certs\") pod \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") "
Oct 31 18:53:47.300601 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.300532    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-maas-api-tls\") pod \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\" (UID: \"bd67e094-084a-4c09-bc37-1ec4a40cc2da\") "
Oct 31 18:53:47.300601 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.300570    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd67e094-084a-4c09-bc37-1ec4a40cc2da-openshift-service-ca" pod "bd67e094-084a-4c09-bc37-1ec4a40cc2da" (UID: "bd67e094-084a-4c09-bc37-1ec4a40cc2da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:53:47.301036 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.301010    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd67e094-084a-4c09-bc37-1ec4a40cc2da-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:47.303607 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.303583    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd67e094-084a-4c09-bc37-1ec4a40cc2da-kube-api-access-7ks4c" pod "bd67e094-084a-4c09-bc37-1ec4a40cc2da" (UID: "bd67e094-084a-4c09-bc37-1ec4a40cc2da"). InnerVolumeSpecName "kube-api-access-7ks4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:53:47.304929 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.303848    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-metrics-certs" pod "bd67e094-084a-4c09-bc37-1ec4a40cc2da" (UID: "bd67e094-084a-4c09-bc37-1ec4a40cc2da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:53:47.304929 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.304012    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-maas-api-tls" pod "bd67e094-084a-4c09-bc37-1ec4a40cc2da" (UID: "bd67e094-084a-4c09-bc37-1ec4a40cc2da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:53:47.380375 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.380338    4484 generic.go:356] "Generic (PLEG): container finished" podID="bd67e094-084a-4c09-bc37-1ec4a40cc2da" containerID="6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589" exitCode=0
Oct 31 18:53:47.380597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.380388    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"
Oct 31 18:53:47.380597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.380432    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r" event={"ID":"bd67e094-084a-4c09-bc37-1ec4a40cc2da","Type":"ContainerDied","Data":"6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589"}
Oct 31 18:53:47.380597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.380520    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r" event={"ID":"bd67e094-084a-4c09-bc37-1ec4a40cc2da","Type":"ContainerDied","Data":"6715c06e0f48b98e8a3b30617824b405f9d5954b9758518e62b43f03cd725f55"}
Oct 31 18:53:47.380597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.380551    4484 scope.go:122] "RemoveContainer" containerID="6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589"
Oct 31 18:53:47.391319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.391303    4484 scope.go:122] "RemoveContainer" containerID="6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589"
Oct 31 18:53:47.391598 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:53:47.391578    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589\": container with ID starting with 6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589 not found: ID does not exist" containerID="6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589"
Oct 31 18:53:47.391657 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.391606    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589"} err="failed to get container status \"6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589\": rpc error: code = NotFound desc = could not find container \"6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589\": container with ID starting with 6a6d0d98edefc48350aee23830f0fca49502245ba7b2c0c125806c3596f1a589 not found: ID does not exist"
Oct 31 18:53:47.402319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.402290    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:47.402319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.402317    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ks4c\" (UniqueName: \"kubernetes.io/projected/bd67e094-084a-4c09-bc37-1ec4a40cc2da-kube-api-access-7ks4c\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:47.402465 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.402332    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd67e094-084a-4c09-bc37-1ec4a40cc2da-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:53:47.545143 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.545109    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"]
Oct 31 18:53:47.552211 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.552187    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-100faf-55658c47d6-h752r"]
Oct 31 18:53:47.994514 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:47.994456    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd67e094-084a-4c09-bc37-1ec4a40cc2da" path="/var/lib/kubelet/pods/bd67e094-084a-4c09-bc37-1ec4a40cc2da/volumes"
Oct 31 18:53:48.769808 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:48.769772    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:48.928191 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:48.928160    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:50.358197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:50.358165    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:50.897113 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:50.897067    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:51.357217 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:51.357182    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:52.418096 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:52.418050    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:53.104389 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:53.104357    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:53.409564 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:53.409464    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:53.900545 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:53.900515    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:55.472415 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:53:55.472373    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:53:55.789215 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:55.789178    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:57.147204 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:57.147175    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:57.327199 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:57.327165    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:57.504088 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:57.504059    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:53:59.271938 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:53:59.271905    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:00.338544 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.338511    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:00.553340 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.553287    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:00.748728 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.748694    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"]
Oct 31 18:54:00.749258 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749246    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f62830c2-01ae-4f6d-acb2-5ab602d17919" containerName="payload-pre-processing"
Oct 31 18:54:00.749314 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749261    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62830c2-01ae-4f6d-acb2-5ab602d17919" containerName="payload-pre-processing"
Oct 31 18:54:00.749314 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749271    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a4415ab7-0e82-435f-b701-544af2b06359" containerName="istio-proxy"
Oct 31 18:54:00.749314 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749276    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4415ab7-0e82-435f-b701-544af2b06359" containerName="istio-proxy"
Oct 31 18:54:00.749314 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749292    4484 cpu_manager.go:414] "containerMap: removing container" podUID="76199531-8025-41d1-813a-cc1a78125e06" containerName="istio-proxy"
Oct 31 18:54:00.749314 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749297    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76199531-8025-41d1-813a-cc1a78125e06" containerName="istio-proxy"
Oct 31 18:54:00.749314 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749314    4484 cpu_manager.go:414] "containerMap: removing container" podUID="416036a1-61e2-4b66-92f7-3236cedbfcd4" containerName="payload-pre-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749319    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416036a1-61e2-4b66-92f7-3236cedbfcd4" containerName="payload-pre-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749326    4484 cpu_manager.go:414] "containerMap: removing container" podUID="46cc80b3-2af6-442e-9080-8576b60ee024" containerName="payload-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749331    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46cc80b3-2af6-442e-9080-8576b60ee024" containerName="payload-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749347    4484 cpu_manager.go:414] "containerMap: removing container" podUID="bd67e094-084a-4c09-bc37-1ec4a40cc2da" containerName="maas-api"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749351    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd67e094-084a-4c09-bc37-1ec4a40cc2da" containerName="maas-api"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749358    4484 cpu_manager.go:414] "containerMap: removing container" podUID="dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b" containerName="payload-pre-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749363    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b" containerName="payload-pre-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749372    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ea5c4327-1827-43f0-8f1e-b711df9dacb6" containerName="payload-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749376    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5c4327-1827-43f0-8f1e-b711df9dacb6" containerName="payload-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749450    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="76199531-8025-41d1-813a-cc1a78125e06" containerName="istio-proxy"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749459    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="416036a1-61e2-4b66-92f7-3236cedbfcd4" containerName="payload-pre-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749465    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="46cc80b3-2af6-442e-9080-8576b60ee024" containerName="payload-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749491    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd67e094-084a-4c09-bc37-1ec4a40cc2da" containerName="maas-api"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749497    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4deeea-bdb8-4e9c-bacd-cb0af59eba9b" containerName="payload-pre-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749503    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5c4327-1827-43f0-8f1e-b711df9dacb6" containerName="payload-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749511    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62830c2-01ae-4f6d-acb2-5ab602d17919" containerName="payload-pre-processing"
Oct 31 18:54:00.749554 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.749515    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4415ab7-0e82-435f-b701-544af2b06359" containerName="istio-proxy"
Oct 31 18:54:00.754261 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.754246    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"
Oct 31 18:54:00.859068 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.859038    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-84c46e-dockercfg-fzfgf\""
Oct 31 18:54:00.861827 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.861786    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnrwm\" (UniqueName: \"kubernetes.io/projected/9289a3ed-73c2-4ceb-9e79-56af282bb294-kube-api-access-dnrwm\") pod \"payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67\" (UID: \"9289a3ed-73c2-4ceb-9e79-56af282bb294\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"
Oct 31 18:54:00.862015 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.861995    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9289a3ed-73c2-4ceb-9e79-56af282bb294-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67\" (UID: \"9289a3ed-73c2-4ceb-9e79-56af282bb294\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"
Oct 31 18:54:00.863415 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.863391    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"]
Oct 31 18:54:00.875758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.875727    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-84c46e\""
Oct 31 18:54:00.893578 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.893553    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"]
Oct 31 18:54:00.897584 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.897569    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"
Oct 31 18:54:00.955782 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.955759    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"]
Oct 31 18:54:00.962723 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.962699    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9289a3ed-73c2-4ceb-9e79-56af282bb294-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67\" (UID: \"9289a3ed-73c2-4ceb-9e79-56af282bb294\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"
Oct 31 18:54:00.962856 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.962744    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr2km\" (UniqueName: \"kubernetes.io/projected/a63f4e31-36d7-4973-a471-8eba39524690-kube-api-access-vr2km\") pod \"payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc\" (UID: \"a63f4e31-36d7-4973-a471-8eba39524690\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"
Oct 31 18:54:00.962856 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.962806    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a63f4e31-36d7-4973-a471-8eba39524690-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc\" (UID: \"a63f4e31-36d7-4973-a471-8eba39524690\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"
Oct 31 18:54:00.962856 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.962844    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnrwm\" (UniqueName: \"kubernetes.io/projected/9289a3ed-73c2-4ceb-9e79-56af282bb294-kube-api-access-dnrwm\") pod \"payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67\" (UID: \"9289a3ed-73c2-4ceb-9e79-56af282bb294\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"
Oct 31 18:54:00.963453 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:00.963433    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9289a3ed-73c2-4ceb-9e79-56af282bb294-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67\" (UID: \"9289a3ed-73c2-4ceb-9e79-56af282bb294\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"
Oct 31 18:54:01.021076 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:01.021000    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnrwm\" (UniqueName: \"kubernetes.io/projected/9289a3ed-73c2-4ceb-9e79-56af282bb294-kube-api-access-dnrwm\") pod \"payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67\" (UID: \"9289a3ed-73c2-4ceb-9e79-56af282bb294\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"
Oct 31 18:54:01.063895 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:01.063869    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr2km\" (UniqueName: \"kubernetes.io/projected/a63f4e31-36d7-4973-a471-8eba39524690-kube-api-access-vr2km\") pod \"payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc\" (UID: \"a63f4e31-36d7-4973-a471-8eba39524690\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"
Oct 31 18:54:01.064024 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:01.063925    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a63f4e31-36d7-4973-a471-8eba39524690-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc\" (UID: \"a63f4e31-36d7-4973-a471-8eba39524690\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"
Oct 31 18:54:01.064653 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:01.064636    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a63f4e31-36d7-4973-a471-8eba39524690-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc\" (UID: \"a63f4e31-36d7-4973-a471-8eba39524690\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"
Oct 31 18:54:01.067608 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:01.067591    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"
Oct 31 18:54:01.122054 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:01.122025    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr2km\" (UniqueName: \"kubernetes.io/projected/a63f4e31-36d7-4973-a471-8eba39524690-kube-api-access-vr2km\") pod \"payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc\" (UID: \"a63f4e31-36d7-4973-a471-8eba39524690\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"
Oct 31 18:54:01.211908 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:01.211881    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"
Oct 31 18:54:01.439385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:01.439358    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"]
Oct 31 18:54:01.440157 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:54:01.440130    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9289a3ed_73c2_4ceb_9e79_56af282bb294.slice/crio-e62bec0910e1331039c907b1381c8d6d3ab8284640279c59301c4f524ccbaba3 WatchSource:0}: Error finding container e62bec0910e1331039c907b1381c8d6d3ab8284640279c59301c4f524ccbaba3: Status 404 returned error can't find the container with id e62bec0910e1331039c907b1381c8d6d3ab8284640279c59301c4f524ccbaba3
Oct 31 18:54:01.491534 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:01.491508    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67" event={"ID":"9289a3ed-73c2-4ceb-9e79-56af282bb294","Type":"ContainerStarted","Data":"e62bec0910e1331039c907b1381c8d6d3ab8284640279c59301c4f524ccbaba3"}
Oct 31 18:54:01.587599 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:01.587577    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"]
Oct 31 18:54:01.588467 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:54:01.588445    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63f4e31_36d7_4973_a471_8eba39524690.slice/crio-8813b728701940a55fbc1f59573d470315548bfaf6a820a05f7c31cb2ae11dd2 WatchSource:0}: Error finding container 8813b728701940a55fbc1f59573d470315548bfaf6a820a05f7c31cb2ae11dd2: Status 404 returned error can't find the container with id 8813b728701940a55fbc1f59573d470315548bfaf6a820a05f7c31cb2ae11dd2
Oct 31 18:54:02.507371 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:02.507323    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc" event={"ID":"a63f4e31-36d7-4973-a471-8eba39524690","Type":"ContainerStarted","Data":"52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1"}
Oct 31 18:54:02.507861 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:02.507559    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc" event={"ID":"a63f4e31-36d7-4973-a471-8eba39524690","Type":"ContainerStarted","Data":"8813b728701940a55fbc1f59573d470315548bfaf6a820a05f7c31cb2ae11dd2"}
Oct 31 18:54:02.507861 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:02.507593    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"
Oct 31 18:54:02.509122 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:02.509099    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67" event={"ID":"9289a3ed-73c2-4ceb-9e79-56af282bb294","Type":"ContainerStarted","Data":"a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd"}
Oct 31 18:54:02.509431 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:02.509222    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"
Oct 31 18:54:02.624656 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:02.624608    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc" podStartSLOduration=2.6245938410000003 podStartE2EDuration="2.624593841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:54:02.621035112 +0000 UTC m=+1988.767484978" watchObservedRunningTime="2026-10-31 18:54:02.624593841 +0000 UTC m=+1988.771043713"
Oct 31 18:54:02.735912 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:02.735858    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67" podStartSLOduration=2.73584385 podStartE2EDuration="2.73584385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:54:02.733963194 +0000 UTC m=+1988.880413084" watchObservedRunningTime="2026-10-31 18:54:02.73584385 +0000 UTC m=+1988.882293780"
Oct 31 18:54:03.030170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:03.030138    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:03.504050 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:03.504011    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:04.594839 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:04.594807    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"]
Oct 31 18:54:04.599062 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:04.599043    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"
Oct 31 18:54:04.654447 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:04.654420    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"]
Oct 31 18:54:04.701023 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:04.700996    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-335e78\""
Oct 31 18:54:04.701023 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:04.701004    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-335e78-dockercfg-8pzc5\""
Oct 31 18:54:04.705168 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:04.705143    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnldv\" (UniqueName: \"kubernetes.io/projected/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-kube-api-access-pnldv\") pod \"payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv\" (UID: \"1e18c6d0-893d-4a0c-a021-e8354d7be8e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"
Oct 31 18:54:04.705323 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:04.705252    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv\" (UID: \"1e18c6d0-893d-4a0c-a021-e8354d7be8e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"
Oct 31 18:54:04.806742 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:04.806713    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnldv\" (UniqueName: \"kubernetes.io/projected/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-kube-api-access-pnldv\") pod \"payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv\" (UID: \"1e18c6d0-893d-4a0c-a021-e8354d7be8e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"
Oct 31 18:54:04.806895 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:04.806779    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv\" (UID: \"1e18c6d0-893d-4a0c-a021-e8354d7be8e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"
Oct 31 18:54:04.807455 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:04.807430    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv\" (UID: \"1e18c6d0-893d-4a0c-a021-e8354d7be8e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"
Oct 31 18:54:04.867279 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:04.867216    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnldv\" (UniqueName: \"kubernetes.io/projected/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-kube-api-access-pnldv\") pod \"payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv\" (UID: \"1e18c6d0-893d-4a0c-a021-e8354d7be8e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"
Oct 31 18:54:04.913592 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:04.913570    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"
Oct 31 18:54:05.288396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.288368    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"]
Oct 31 18:54:05.289169 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:54:05.289149    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e18c6d0_893d_4a0c_a021_e8354d7be8e5.slice/crio-ee841f440dfe6b61786a45a39aa23a47f199ed7b04425f7e714518aef1e10e9b WatchSource:0}: Error finding container ee841f440dfe6b61786a45a39aa23a47f199ed7b04425f7e714518aef1e10e9b: Status 404 returned error can't find the container with id ee841f440dfe6b61786a45a39aa23a47f199ed7b04425f7e714518aef1e10e9b
Oct 31 18:54:05.394424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.394393    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"]
Oct 31 18:54:05.401250 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.401212    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"
Oct 31 18:54:05.453018 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.452987    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"]
Oct 31 18:54:05.515335 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.515306    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96bd01d-ee97-4e78-8171-df82fe0513c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh\" (UID: \"f96bd01d-ee97-4e78-8171-df82fe0513c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"
Oct 31 18:54:05.515511 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.515380    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8l7r\" (UniqueName: \"kubernetes.io/projected/f96bd01d-ee97-4e78-8171-df82fe0513c5-kube-api-access-l8l7r\") pod \"payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh\" (UID: \"f96bd01d-ee97-4e78-8171-df82fe0513c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"
Oct 31 18:54:05.529279 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.529242    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv" event={"ID":"1e18c6d0-893d-4a0c-a021-e8354d7be8e5","Type":"ContainerStarted","Data":"83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d"}
Oct 31 18:54:05.529279 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.529284    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv" event={"ID":"1e18c6d0-893d-4a0c-a021-e8354d7be8e5","Type":"ContainerStarted","Data":"ee841f440dfe6b61786a45a39aa23a47f199ed7b04425f7e714518aef1e10e9b"}
Oct 31 18:54:05.529519 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.529327    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"
Oct 31 18:54:05.617029 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.616956    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96bd01d-ee97-4e78-8171-df82fe0513c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh\" (UID: \"f96bd01d-ee97-4e78-8171-df82fe0513c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"
Oct 31 18:54:05.617468 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.617051    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8l7r\" (UniqueName: \"kubernetes.io/projected/f96bd01d-ee97-4e78-8171-df82fe0513c5-kube-api-access-l8l7r\") pod \"payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh\" (UID: \"f96bd01d-ee97-4e78-8171-df82fe0513c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"
Oct 31 18:54:05.617760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.617742    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96bd01d-ee97-4e78-8171-df82fe0513c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh\" (UID: \"f96bd01d-ee97-4e78-8171-df82fe0513c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"
Oct 31 18:54:05.645230 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.645188    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv" podStartSLOduration=1.6451775039999998 podStartE2EDuration="1.645177504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:54:05.642884984 +0000 UTC m=+1991.789334850" watchObservedRunningTime="2026-10-31 18:54:05.645177504 +0000 UTC m=+1991.791627436"
Oct 31 18:54:05.674243 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.674216    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8l7r\" (UniqueName: \"kubernetes.io/projected/f96bd01d-ee97-4e78-8171-df82fe0513c5-kube-api-access-l8l7r\") pod \"payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh\" (UID: \"f96bd01d-ee97-4e78-8171-df82fe0513c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"
Oct 31 18:54:05.716207 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:05.716169    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"
Oct 31 18:54:06.107970 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:06.107940    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"]
Oct 31 18:54:06.108626 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:54:06.108601    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96bd01d_ee97_4e78_8171_df82fe0513c5.slice/crio-b587067d0a0b98cfa1e3c3f73da4c1f26ebc370f1ff2c6fd26aff77adb006f34 WatchSource:0}: Error finding container b587067d0a0b98cfa1e3c3f73da4c1f26ebc370f1ff2c6fd26aff77adb006f34: Status 404 returned error can't find the container with id b587067d0a0b98cfa1e3c3f73da4c1f26ebc370f1ff2c6fd26aff77adb006f34
Oct 31 18:54:06.175794 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:06.175764    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:06.538176 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:06.538139    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh" event={"ID":"f96bd01d-ee97-4e78-8171-df82fe0513c5","Type":"ContainerStarted","Data":"ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d"}
Oct 31 18:54:06.538329 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:06.538200    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh" event={"ID":"f96bd01d-ee97-4e78-8171-df82fe0513c5","Type":"ContainerStarted","Data":"b587067d0a0b98cfa1e3c3f73da4c1f26ebc370f1ff2c6fd26aff77adb006f34"}
Oct 31 18:54:06.655265 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:06.655209    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh" podStartSLOduration=1.655190942 podStartE2EDuration="1.655190942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:54:06.653248614 +0000 UTC m=+1992.799698479" watchObservedRunningTime="2026-10-31 18:54:06.655190942 +0000 UTC m=+1992.801640810"
Oct 31 18:54:07.543761 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:07.543729    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"
Oct 31 18:54:12.336580 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:12.336535    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"]
Oct 31 18:54:12.337051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:12.336777    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" podUID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" containerName="istio-proxy" containerID="cri-o://e2b317cae9f74f8b500dfc898e7dd599533c936ad970c57c6019a1180112f52c" gracePeriod=30
Oct 31 18:54:12.565864 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:12.565834    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:13.518324 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:13.518297    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"
Oct 31 18:54:13.518720 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:13.518361    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"
Oct 31 18:54:14.581513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:14.581455    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wxkqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:54:14.581899 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:14.581534    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" podUID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:54:16.540907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:16.540877    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"
Oct 31 18:54:17.438830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.438798    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:17.606755 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.606688    4484 generic.go:356] "Generic (PLEG): container finished" podID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" containerID="e2b317cae9f74f8b500dfc898e7dd599533c936ad970c57c6019a1180112f52c" exitCode=0
Oct 31 18:54:17.606755 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.606733    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:54:17.607095 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.606754    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" event={"ID":"3ab39de0-0e3b-489f-91b9-3703f4957aa2","Type":"ContainerDied","Data":"e2b317cae9f74f8b500dfc898e7dd599533c936ad970c57c6019a1180112f52c"}
Oct 31 18:54:17.607095 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.606805    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh" event={"ID":"3ab39de0-0e3b-489f-91b9-3703f4957aa2","Type":"ContainerDied","Data":"bb731c1535c60142637e13ab13d6380aad03d39bb2758944d9f7a361288073ee"}
Oct 31 18:54:17.607095 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.606825    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb731c1535c60142637e13ab13d6380aad03d39bb2758944d9f7a361288073ee"
Oct 31 18:54:17.663721 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.663690    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-certs\") pod \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") "
Oct 31 18:54:17.663889 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.663787    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istiod-ca-cert\") pod \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") "
Oct 31 18:54:17.663889 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.663862    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-socket\") pod \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") "
Oct 31 18:54:17.663981 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.663888    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-kube-api-access-w85cv\" (UniqueName: \"kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-kube-api-access-w85cv\") pod \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") "
Oct 31 18:54:17.663981 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.663937    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-envoy\") pod \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") "
Oct 31 18:54:17.663981 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.663958    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-data\") pod \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") "
Oct 31 18:54:17.664135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.663977    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-certs" pod "3ab39de0-0e3b-489f-91b9-3703f4957aa2" (UID: "3ab39de0-0e3b-489f-91b9-3703f4957aa2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:54:17.664135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.663987    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-token\" (UniqueName: \"kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-token\") pod \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") "
Oct 31 18:54:17.664135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.664065    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-podinfo\") pod \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") "
Oct 31 18:54:17.664135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.664107    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-credential-socket\") pod \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\" (UID: \"3ab39de0-0e3b-489f-91b9-3703f4957aa2\") "
Oct 31 18:54:17.664135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.664114    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-socket" pod "3ab39de0-0e3b-489f-91b9-3703f4957aa2" (UID: "3ab39de0-0e3b-489f-91b9-3703f4957aa2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:54:17.664358 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.664167    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istiod-ca-cert" pod "3ab39de0-0e3b-489f-91b9-3703f4957aa2" (UID: "3ab39de0-0e3b-489f-91b9-3703f4957aa2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:54:17.664464 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.664441    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-data" pod "3ab39de0-0e3b-489f-91b9-3703f4957aa2" (UID: "3ab39de0-0e3b-489f-91b9-3703f4957aa2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:54:17.664658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.664630    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-credential-socket" pod "3ab39de0-0e3b-489f-91b9-3703f4957aa2" (UID: "3ab39de0-0e3b-489f-91b9-3703f4957aa2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:54:17.664742 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.664724    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:17.664801 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.664744    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:17.664801 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.664759    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:17.664801 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.664773    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:17.664801 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.664790    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:17.666337 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.666310    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-kube-api-access-w85cv" pod "3ab39de0-0e3b-489f-91b9-3703f4957aa2" (UID: "3ab39de0-0e3b-489f-91b9-3703f4957aa2"). InnerVolumeSpecName "kube-api-access-w85cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:54:17.666337 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.666323    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-token" pod "3ab39de0-0e3b-489f-91b9-3703f4957aa2" (UID: "3ab39de0-0e3b-489f-91b9-3703f4957aa2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:54:17.666493 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.666370    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-envoy" pod "3ab39de0-0e3b-489f-91b9-3703f4957aa2" (UID: "3ab39de0-0e3b-489f-91b9-3703f4957aa2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:54:17.666531 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.666464    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-podinfo" pod "3ab39de0-0e3b-489f-91b9-3703f4957aa2" (UID: "3ab39de0-0e3b-489f-91b9-3703f4957aa2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:54:17.766245 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.766217    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w85cv\" (UniqueName: \"kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-kube-api-access-w85cv\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:17.766245 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.766241    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:17.766245 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.766250    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:17.766455 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:17.766259    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ab39de0-0e3b-489f-91b9-3703f4957aa2-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:54:18.551622 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:18.551590    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"
Oct 31 18:54:18.612032 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:18.612004    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"
Oct 31 18:54:18.779872 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:18.779840    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"]
Oct 31 18:54:18.829047 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:18.828981    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wxkqh"]
Oct 31 18:54:19.993968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:19.993881    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" path="/var/lib/kubelet/pods/3ab39de0-0e3b-489f-91b9-3703f4957aa2/volumes"
Oct 31 18:54:20.906948 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:20.906912    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:21.388892 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:21.388851    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:22.477397 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.477361    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"]
Oct 31 18:54:22.478194 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.478172    4484 cpu_manager.go:414] "containerMap: removing container" podUID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" containerName="istio-proxy"
Oct 31 18:54:22.478298 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.478197    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" containerName="istio-proxy"
Oct 31 18:54:22.478382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.478368    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab39de0-0e3b-489f-91b9-3703f4957aa2" containerName="istio-proxy"
Oct 31 18:54:22.482855 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.482835    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"
Oct 31 18:54:22.538141 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.538116    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"]
Oct 31 18:54:22.585156 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.585130    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ceef3b\""
Oct 31 18:54:22.585320 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.585202    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ceef3b-dockercfg-7f9cp\""
Oct 31 18:54:22.625049 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.625021    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2cbp\" (UniqueName: \"kubernetes.io/projected/c56988bf-4c50-48b4-b276-b3501f735545-kube-api-access-f2cbp\") pod \"payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59\" (UID: \"c56988bf-4c50-48b4-b276-b3501f735545\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"
Oct 31 18:54:22.625170 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.625070    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56988bf-4c50-48b4-b276-b3501f735545-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59\" (UID: \"c56988bf-4c50-48b4-b276-b3501f735545\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"
Oct 31 18:54:22.726070 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.726042    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2cbp\" (UniqueName: \"kubernetes.io/projected/c56988bf-4c50-48b4-b276-b3501f735545-kube-api-access-f2cbp\") pod \"payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59\" (UID: \"c56988bf-4c50-48b4-b276-b3501f735545\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"
Oct 31 18:54:22.726218 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.726094    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56988bf-4c50-48b4-b276-b3501f735545-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59\" (UID: \"c56988bf-4c50-48b4-b276-b3501f735545\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"
Oct 31 18:54:22.726780 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.726758    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56988bf-4c50-48b4-b276-b3501f735545-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59\" (UID: \"c56988bf-4c50-48b4-b276-b3501f735545\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"
Oct 31 18:54:22.783278 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.783253    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2cbp\" (UniqueName: \"kubernetes.io/projected/c56988bf-4c50-48b4-b276-b3501f735545-kube-api-access-f2cbp\") pod \"payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59\" (UID: \"c56988bf-4c50-48b4-b276-b3501f735545\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"
Oct 31 18:54:22.796147 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:22.796126    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"
Oct 31 18:54:23.169505 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.169460    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"]
Oct 31 18:54:23.170086 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:54:23.170059    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56988bf_4c50_48b4_b276_b3501f735545.slice/crio-3fc6dcbb20e7a84239c64f9442b0ea2acc8320383898f42be0876f4b69780bb1 WatchSource:0}: Error finding container 3fc6dcbb20e7a84239c64f9442b0ea2acc8320383898f42be0876f4b69780bb1: Status 404 returned error can't find the container with id 3fc6dcbb20e7a84239c64f9442b0ea2acc8320383898f42be0876f4b69780bb1
Oct 31 18:54:23.272514 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.272484    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"]
Oct 31 18:54:23.276648 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.276627    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"
Oct 31 18:54:23.334231 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.334200    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"]
Oct 31 18:54:23.336881 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.336853    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwx6r\" (UniqueName: \"kubernetes.io/projected/05c4f0a0-e452-4fb3-b72c-5dda7639c882-kube-api-access-bwx6r\") pod \"payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm\" (UID: \"05c4f0a0-e452-4fb3-b72c-5dda7639c882\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"
Oct 31 18:54:23.336987 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.336966    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c4f0a0-e452-4fb3-b72c-5dda7639c882-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm\" (UID: \"05c4f0a0-e452-4fb3-b72c-5dda7639c882\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"
Oct 31 18:54:23.437932 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.437865    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwx6r\" (UniqueName: \"kubernetes.io/projected/05c4f0a0-e452-4fb3-b72c-5dda7639c882-kube-api-access-bwx6r\") pod \"payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm\" (UID: \"05c4f0a0-e452-4fb3-b72c-5dda7639c882\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"
Oct 31 18:54:23.437932 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.437920    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c4f0a0-e452-4fb3-b72c-5dda7639c882-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm\" (UID: \"05c4f0a0-e452-4fb3-b72c-5dda7639c882\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"
Oct 31 18:54:23.438833 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.438809    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c4f0a0-e452-4fb3-b72c-5dda7639c882-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm\" (UID: \"05c4f0a0-e452-4fb3-b72c-5dda7639c882\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"
Oct 31 18:54:23.495508 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.495458    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwx6r\" (UniqueName: \"kubernetes.io/projected/05c4f0a0-e452-4fb3-b72c-5dda7639c882-kube-api-access-bwx6r\") pod \"payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm\" (UID: \"05c4f0a0-e452-4fb3-b72c-5dda7639c882\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"
Oct 31 18:54:23.592633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.592605    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"
Oct 31 18:54:23.644067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.644033    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59" event={"ID":"c56988bf-4c50-48b4-b276-b3501f735545","Type":"ContainerStarted","Data":"41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489"}
Oct 31 18:54:23.644214 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.644087    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59" event={"ID":"c56988bf-4c50-48b4-b276-b3501f735545","Type":"ContainerStarted","Data":"3fc6dcbb20e7a84239c64f9442b0ea2acc8320383898f42be0876f4b69780bb1"}
Oct 31 18:54:23.644214 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.644141    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"
Oct 31 18:54:23.760816 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.760767    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59" podStartSLOduration=1.760755025 podStartE2EDuration="1.760755025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:54:23.757539837 +0000 UTC m=+2009.903989702" watchObservedRunningTime="2026-10-31 18:54:23.760755025 +0000 UTC m=+2009.907204915"
Oct 31 18:54:23.958950 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.958922    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:54:23.968835 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.968811    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"]
Oct 31 18:54:23.969119 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:54:23.969082    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c4f0a0_e452_4fb3_b72c_5dda7639c882.slice/crio-b9e4beb2a7727e4df34a1c1aaa1f096cb78ac3cab4bc60f13cfdfaf057fd8955 WatchSource:0}: Error finding container b9e4beb2a7727e4df34a1c1aaa1f096cb78ac3cab4bc60f13cfdfaf057fd8955: Status 404 returned error can't find the container with id b9e4beb2a7727e4df34a1c1aaa1f096cb78ac3cab4bc60f13cfdfaf057fd8955
Oct 31 18:54:23.973252 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:23.973215    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:24.651431 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:24.651401    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm" event={"ID":"05c4f0a0-e452-4fb3-b72c-5dda7639c882","Type":"ContainerStarted","Data":"1a1d30a3674b61c59d3d888a890b2e41055f77d770b33ba36ee079e85485b56e"}
Oct 31 18:54:24.651926 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:24.651460    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm" event={"ID":"05c4f0a0-e452-4fb3-b72c-5dda7639c882","Type":"ContainerStarted","Data":"b9e4beb2a7727e4df34a1c1aaa1f096cb78ac3cab4bc60f13cfdfaf057fd8955"}
Oct 31 18:54:24.651926 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:24.651725    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"
Oct 31 18:54:24.766830 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:24.766789    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm" podStartSLOduration=1.766774795 podStartE2EDuration="1.766774795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:54:24.765698877 +0000 UTC m=+2010.912148746" watchObservedRunningTime="2026-10-31 18:54:24.766774795 +0000 UTC m=+2010.913224713"
Oct 31 18:54:25.219975 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:25.219932    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:25.468586 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:54:25.468551    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:54:28.749856 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:28.749822    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:31.634803 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:31.634770    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:34.653678 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:34.653650    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"
Oct 31 18:54:35.534875 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:35.534838    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:35.660460 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:35.660398    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"
Oct 31 18:54:39.378339 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:39.378298    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:39.851849 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:39.851810    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:40.204419 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:40.204332    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:41.739097 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:41.739065    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"]
Oct 31 18:54:41.744460 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:41.744439    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"
Oct 31 18:54:41.799060 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:41.799023    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"]
Oct 31 18:54:41.845130 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:41.845100    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-676224\""
Oct 31 18:54:41.845130 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:41.845124    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-676224-dockercfg-rqsqk\""
Oct 31 18:54:41.850298 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:41.850276    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l882\" (UniqueName: \"kubernetes.io/projected/16186434-2291-4ba4-bd98-ddcfa937eff6-kube-api-access-9l882\") pod \"payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp\" (UID: \"16186434-2291-4ba4-bd98-ddcfa937eff6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"
Oct 31 18:54:41.850431 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:41.850415    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16186434-2291-4ba4-bd98-ddcfa937eff6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp\" (UID: \"16186434-2291-4ba4-bd98-ddcfa937eff6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"
Oct 31 18:54:41.951492 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:41.951435    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l882\" (UniqueName: \"kubernetes.io/projected/16186434-2291-4ba4-bd98-ddcfa937eff6-kube-api-access-9l882\") pod \"payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp\" (UID: \"16186434-2291-4ba4-bd98-ddcfa937eff6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"
Oct 31 18:54:41.951649 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:41.951574    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16186434-2291-4ba4-bd98-ddcfa937eff6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp\" (UID: \"16186434-2291-4ba4-bd98-ddcfa937eff6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"
Oct 31 18:54:41.952190 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:41.952173    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16186434-2291-4ba4-bd98-ddcfa937eff6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp\" (UID: \"16186434-2291-4ba4-bd98-ddcfa937eff6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"
Oct 31 18:54:42.008843 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:42.008771    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l882\" (UniqueName: \"kubernetes.io/projected/16186434-2291-4ba4-bd98-ddcfa937eff6-kube-api-access-9l882\") pod \"payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp\" (UID: \"16186434-2291-4ba4-bd98-ddcfa937eff6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"
Oct 31 18:54:42.058804 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:42.058779    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"
Oct 31 18:54:42.433423 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:42.433400    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"]
Oct 31 18:54:42.434293 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:54:42.434265    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16186434_2291_4ba4_bd98_ddcfa937eff6.slice/crio-585f11662ff0c1f2677e2ae8cbb063ec240197bd90acf090e9acc56a5d32f38e WatchSource:0}: Error finding container 585f11662ff0c1f2677e2ae8cbb063ec240197bd90acf090e9acc56a5d32f38e: Status 404 returned error can't find the container with id 585f11662ff0c1f2677e2ae8cbb063ec240197bd90acf090e9acc56a5d32f38e
Oct 31 18:54:42.536974 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:42.536928    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:42.768362 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:42.768327    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp" event={"ID":"16186434-2291-4ba4-bd98-ddcfa937eff6","Type":"ContainerStarted","Data":"9240940e9856c7be2b4d16ab172c57f1d45b936cb2267e2fc3b318e56d4587b3"}
Oct 31 18:54:42.768762 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:42.768386    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp" event={"ID":"16186434-2291-4ba4-bd98-ddcfa937eff6","Type":"ContainerStarted","Data":"585f11662ff0c1f2677e2ae8cbb063ec240197bd90acf090e9acc56a5d32f38e"}
Oct 31 18:54:42.768762 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:42.768624    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"
Oct 31 18:54:42.885838 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:42.885786    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp" podStartSLOduration=1.885772681 podStartE2EDuration="1.885772681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:54:42.882864717 +0000 UTC m=+2029.029314582" watchObservedRunningTime="2026-10-31 18:54:42.885772681 +0000 UTC m=+2029.032222544"
Oct 31 18:54:46.161626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:46.161587    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:47.362444 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:47.362392    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:51.689553 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:51.689512    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:52.460229 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:52.460198    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:53.777038 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:53.777005    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"
Oct 31 18:54:54.770909 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:54.770880    4484 scope.go:122] "RemoveContainer" containerID="e2b317cae9f74f8b500dfc898e7dd599533c936ad970c57c6019a1180112f52c"
Oct 31 18:54:55.467985 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:54:55.467947    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:54:55.505067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:55.505024    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:56.169622 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:56.169588    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:56.636531 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:56.636467    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:56.841894 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:56.841862    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:54:57.102460 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:54:57.102419    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:00.615565 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:00.615534    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:00.776802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:00.776769    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:00.951181 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:00.951101    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:02.039924 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:02.039897    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:04.662868 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:04.662836    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:07.056627 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:07.056589    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:07.211586 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:07.211550    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:07.362739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:07.362658    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:07.513765 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:07.513733    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:11.609664 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:11.609629    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:11.805876 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:11.805838    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:11.970238 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:11.970152    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:12.853409 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:12.853370    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:13.475307 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:13.475272    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"]
Oct 31 18:55:13.853580 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:13.853543    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:14.057136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:14.057097    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:16.957093 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:16.957054    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:18.374689 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.374654    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:18.476396 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.476359    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n" podUID="74824d04-008e-4e17-8ff6-1060a8a990a2" containerName="payload-processing" containerID="cri-o://86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969" gracePeriod=25
Oct 31 18:55:18.738404 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.738381    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"
Oct 31 18:55:18.867950 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.867913    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74824d04-008e-4e17-8ff6-1060a8a990a2-kube-api-access-xdvlq\" (UniqueName: \"kubernetes.io/projected/74824d04-008e-4e17-8ff6-1060a8a990a2-kube-api-access-xdvlq\") pod \"74824d04-008e-4e17-8ff6-1060a8a990a2\" (UID: \"74824d04-008e-4e17-8ff6-1060a8a990a2\") "
Oct 31 18:55:18.868181 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.868161    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74824d04-008e-4e17-8ff6-1060a8a990a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74824d04-008e-4e17-8ff6-1060a8a990a2-plugins-config-volume\") pod \"74824d04-008e-4e17-8ff6-1060a8a990a2\" (UID: \"74824d04-008e-4e17-8ff6-1060a8a990a2\") "
Oct 31 18:55:18.868613 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.868586    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74824d04-008e-4e17-8ff6-1060a8a990a2-plugins-config-volume" pod "74824d04-008e-4e17-8ff6-1060a8a990a2" (UID: "74824d04-008e-4e17-8ff6-1060a8a990a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:55:18.868897 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.868876    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74824d04-008e-4e17-8ff6-1060a8a990a2-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:18.870651 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.870625    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74824d04-008e-4e17-8ff6-1060a8a990a2-kube-api-access-xdvlq" pod "74824d04-008e-4e17-8ff6-1060a8a990a2" (UID: "74824d04-008e-4e17-8ff6-1060a8a990a2"). InnerVolumeSpecName "kube-api-access-xdvlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:18.970231 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.970143    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdvlq\" (UniqueName: \"kubernetes.io/projected/74824d04-008e-4e17-8ff6-1060a8a990a2-kube-api-access-xdvlq\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:18.996662 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.996628    4484 generic.go:356] "Generic (PLEG): container finished" podID="74824d04-008e-4e17-8ff6-1060a8a990a2" containerID="86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969" exitCode=0
Oct 31 18:55:18.996836 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.996675    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"
Oct 31 18:55:18.996836 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.996712    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n" event={"ID":"74824d04-008e-4e17-8ff6-1060a8a990a2","Type":"ContainerDied","Data":"86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969"}
Oct 31 18:55:18.996836 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.996766    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n" event={"ID":"74824d04-008e-4e17-8ff6-1060a8a990a2","Type":"ContainerDied","Data":"f065d7a09d4eeda5eb1d6889bd1ac7b4927e228a82385d021375ff02fde7d5ca"}
Oct 31 18:55:18.996836 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:18.996791    4484 scope.go:122] "RemoveContainer" containerID="86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969"
Oct 31 18:55:19.006912 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:19.006891    4484 scope.go:122] "RemoveContainer" containerID="86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969"
Oct 31 18:55:19.007207 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:55:19.007187    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969\": container with ID starting with 86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969 not found: ID does not exist" containerID="86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969"
Oct 31 18:55:19.007260 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:19.007216    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969"} err="failed to get container status \"86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969\": rpc error: code = NotFound desc = could not find container \"86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969\": container with ID starting with 86f300674fa684216fa635fb93860cd914d71eb820da1a663ca966b18c2a9969 not found: ID does not exist"
Oct 31 18:55:19.162985 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:19.162949    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"]
Oct 31 18:55:19.165580 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:19.165561    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-aa4693-9f55f8fff-lrz2n"]
Oct 31 18:55:19.994417 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:19.994386    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74824d04-008e-4e17-8ff6-1060a8a990a2" path="/var/lib/kubelet/pods/74824d04-008e-4e17-8ff6-1060a8a990a2/volumes"
Oct 31 18:55:23.022555 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:23.022521    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:23.339858 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:23.339779    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"]
Oct 31 18:55:23.340035 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:23.340016    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q" podUID="711334f9-d214-4503-bf64-4e0b8bc408a6" containerName="payload-processing" containerID="cri-o://c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88" gracePeriod=30
Oct 31 18:55:23.516944 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:23.516790    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:23.648878 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:23.648852    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"
Oct 31 18:55:23.665637 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:23.665591    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:23.722349 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:23.722321    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711334f9-d214-4503-bf64-4e0b8bc408a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711334f9-d214-4503-bf64-4e0b8bc408a6-plugins-config-volume\") pod \"711334f9-d214-4503-bf64-4e0b8bc408a6\" (UID: \"711334f9-d214-4503-bf64-4e0b8bc408a6\") "
Oct 31 18:55:23.722513 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:23.722499    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711334f9-d214-4503-bf64-4e0b8bc408a6-kube-api-access-pxw6j\" (UniqueName: \"kubernetes.io/projected/711334f9-d214-4503-bf64-4e0b8bc408a6-kube-api-access-pxw6j\") pod \"711334f9-d214-4503-bf64-4e0b8bc408a6\" (UID: \"711334f9-d214-4503-bf64-4e0b8bc408a6\") "
Oct 31 18:55:23.722688 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:23.722667    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711334f9-d214-4503-bf64-4e0b8bc408a6-plugins-config-volume" pod "711334f9-d214-4503-bf64-4e0b8bc408a6" (UID: "711334f9-d214-4503-bf64-4e0b8bc408a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:55:23.722802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:23.722789    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711334f9-d214-4503-bf64-4e0b8bc408a6-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:23.724601 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:23.724580    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711334f9-d214-4503-bf64-4e0b8bc408a6-kube-api-access-pxw6j" pod "711334f9-d214-4503-bf64-4e0b8bc408a6" (UID: "711334f9-d214-4503-bf64-4e0b8bc408a6"). InnerVolumeSpecName "kube-api-access-pxw6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:23.824087 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:23.824061    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxw6j\" (UniqueName: \"kubernetes.io/projected/711334f9-d214-4503-bf64-4e0b8bc408a6-kube-api-access-pxw6j\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:24.032032 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:24.031998    4484 generic.go:356] "Generic (PLEG): container finished" podID="711334f9-d214-4503-bf64-4e0b8bc408a6" containerID="c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88" exitCode=0
Oct 31 18:55:24.032399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:24.032037    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"
Oct 31 18:55:24.032399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:24.032079    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q" event={"ID":"711334f9-d214-4503-bf64-4e0b8bc408a6","Type":"ContainerDied","Data":"c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88"}
Oct 31 18:55:24.032399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:24.032121    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q" event={"ID":"711334f9-d214-4503-bf64-4e0b8bc408a6","Type":"ContainerDied","Data":"1224993ef8cc43884dfc7ed6295cd9580b101366e89d9f409a05cdc08cd82912"}
Oct 31 18:55:24.032399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:24.032144    4484 scope.go:122] "RemoveContainer" containerID="c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88"
Oct 31 18:55:24.041320 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:24.041298    4484 scope.go:122] "RemoveContainer" containerID="c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88"
Oct 31 18:55:24.041622 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:55:24.041605    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88\": container with ID starting with c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88 not found: ID does not exist" containerID="c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88"
Oct 31 18:55:24.041671 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:24.041629    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88"} err="failed to get container status \"c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88\": rpc error: code = NotFound desc = could not find container \"c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88\": container with ID starting with c51f6cdd63ccbfb9e063b720fefff148d740c36877888be2267a7e9fdbd3eb88 not found: ID does not exist"
Oct 31 18:55:24.198167 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:24.198144    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"]
Oct 31 18:55:24.200716 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:24.200693    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a4923-69c9678764-pc47q"]
Oct 31 18:55:24.745759 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:24.745723    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:25.468435 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:55:25.468403    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:55:25.995716 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:25.995687    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711334f9-d214-4503-bf64-4e0b8bc408a6" path="/var/lib/kubelet/pods/711334f9-d214-4503-bf64-4e0b8bc408a6/volumes"
Oct 31 18:55:26.260189 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:26.260105    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:26.517831 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:26.517748    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:26.813385 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:26.813304    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:55:27.893182 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:27.893146    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:28.434179 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:28.434121    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:29.207697 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:29.207661    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:29.792952 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:29.792915    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:31.809168 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:31.809135    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:31.959009 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:31.958973    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:35.088016 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:35.087980    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:36.457997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:36.457963    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:40.671025 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:40.670989    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"]
Oct 31 18:55:40.671443 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:40.671198    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd" podUID="24f9a5e2-42a9-4f4a-9289-8bb7699eb450" containerName="maas-api" containerID="cri-o://74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6" gracePeriod=30
Oct 31 18:55:40.793769 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:40.793733    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:40.939256 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:40.939199    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:55:41.019077 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.019044    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-metrics-certs\") pod \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") "
Oct 31 18:55:41.019246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.019100    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-maas-api-tls\") pod \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") "
Oct 31 18:55:41.019246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.019138    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-openshift-service-ca\") pod \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") "
Oct 31 18:55:41.019246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.019163    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-kube-api-access-q5vtv\" (UniqueName: \"kubernetes.io/projected/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-kube-api-access-q5vtv\") pod \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\" (UID: \"24f9a5e2-42a9-4f4a-9289-8bb7699eb450\") "
Oct 31 18:55:41.021820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.019845    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-openshift-service-ca" pod "24f9a5e2-42a9-4f4a-9289-8bb7699eb450" (UID: "24f9a5e2-42a9-4f4a-9289-8bb7699eb450"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:55:41.021820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.020433    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:41.031581 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.022142    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-metrics-certs" pod "24f9a5e2-42a9-4f4a-9289-8bb7699eb450" (UID: "24f9a5e2-42a9-4f4a-9289-8bb7699eb450"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:55:41.031581 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.022384    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-maas-api-tls" pod "24f9a5e2-42a9-4f4a-9289-8bb7699eb450" (UID: "24f9a5e2-42a9-4f4a-9289-8bb7699eb450"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:55:41.032443 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.032417    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-kube-api-access-q5vtv" pod "24f9a5e2-42a9-4f4a-9289-8bb7699eb450" (UID: "24f9a5e2-42a9-4f4a-9289-8bb7699eb450"). InnerVolumeSpecName "kube-api-access-q5vtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:55:41.121241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.121211    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:41.121241 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.121239    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:41.121449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.121250    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5vtv\" (UniqueName: \"kubernetes.io/projected/24f9a5e2-42a9-4f4a-9289-8bb7699eb450-kube-api-access-q5vtv\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:55:41.143002 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.142970    4484 generic.go:356] "Generic (PLEG): container finished" podID="24f9a5e2-42a9-4f4a-9289-8bb7699eb450" containerID="74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6" exitCode=0
Oct 31 18:55:41.143152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.143051    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"
Oct 31 18:55:41.143152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.143054    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd" event={"ID":"24f9a5e2-42a9-4f4a-9289-8bb7699eb450","Type":"ContainerDied","Data":"74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6"}
Oct 31 18:55:41.143152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.143134    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd" event={"ID":"24f9a5e2-42a9-4f4a-9289-8bb7699eb450","Type":"ContainerDied","Data":"b4687efb21871a650cb614c2997df0b5ad4013f70a114e9cf4c2a1ed6ae6f559"}
Oct 31 18:55:41.143267 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.143158    4484 scope.go:122] "RemoveContainer" containerID="74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6"
Oct 31 18:55:41.153236 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.153219    4484 scope.go:122] "RemoveContainer" containerID="74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6"
Oct 31 18:55:41.153534 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:55:41.153510    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6\": container with ID starting with 74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6 not found: ID does not exist" containerID="74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6"
Oct 31 18:55:41.153637 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.153539    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6"} err="failed to get container status \"74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6\": rpc error: code = NotFound desc = could not find container \"74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6\": container with ID starting with 74caf6cefc31bdfd7838aecd28291bb9e4976a458b187a3369b0ebab473c15a6 not found: ID does not exist"
Oct 31 18:55:41.309622 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.309582    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"]
Oct 31 18:55:41.312802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.312775    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d1b3f-6887f97f4b-glfzd"]
Oct 31 18:55:41.365882 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.365852    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:41.515348 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.515313    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:41.994616 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:41.994587    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f9a5e2-42a9-4f4a-9289-8bb7699eb450" path="/var/lib/kubelet/pods/24f9a5e2-42a9-4f4a-9289-8bb7699eb450/volumes"
Oct 31 18:55:42.864527 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:42.864492    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:47.360636 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:47.360607    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:47.619202 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:47.619110    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:48.926274 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:48.926228    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:49.341778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:49.341737    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:49.516269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:49.516232    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:50.267271 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:50.267232    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:50.633155 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:50.633063    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:50.787625 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:50.787585    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:50.972026 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:50.971943    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:52.025165 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:52.025128    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:52.227236 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:52.227203    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:52.615970 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:52.615942    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:53.409142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:53.409106    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:54.525789 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:54.525759    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:55:54.533500 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:54.533455    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 18:55:54.649893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:54.649865    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:55:55.311954 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:55.311925    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"]
Oct 31 18:55:55.312166 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:55.312142    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n" podUID="ec4eaec1-65e4-4507-b4fd-7938a9adcd87" containerName="istio-proxy" containerID="cri-o://e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c" gracePeriod=30
Oct 31 18:55:55.470869 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:55:55.470835    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:55:56.050966 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:56.050929    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:58.032616 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:58.032584    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:58.966614 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:58.966571    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"]
Oct 31 18:55:59.073451 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:59.073421    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"]
Oct 31 18:55:59.341751 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:59.341718    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:59.595954 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:59.595869    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:55:59.758967 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:55:59.758930    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:00.600420 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.600393    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:56:00.694910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.694880    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:00.743727 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.743688    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-podinfo\") pod \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") "
Oct 31 18:56:00.743913 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.743751    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-kube-api-access-pb4ws\" (UniqueName: \"kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-kube-api-access-pb4ws\") pod \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") "
Oct 31 18:56:00.743913 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.743786    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-token\" (UniqueName: \"kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-token\") pod \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") "
Oct 31 18:56:00.743913 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.743820    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-socket\") pod \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") "
Oct 31 18:56:00.743913 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.743877    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istiod-ca-cert\") pod \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") "
Oct 31 18:56:00.744126 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.743979    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-data\") pod \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") "
Oct 31 18:56:00.744126 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.744095    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-credential-socket\") pod \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") "
Oct 31 18:56:00.744233 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.744134    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-certs\") pod \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") "
Oct 31 18:56:00.744629 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.744484    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-credential-socket" pod "ec4eaec1-65e4-4507-b4fd-7938a9adcd87" (UID: "ec4eaec1-65e4-4507-b4fd-7938a9adcd87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:00.744758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.744635    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-envoy\") pod \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\" (UID: \"ec4eaec1-65e4-4507-b4fd-7938a9adcd87\") "
Oct 31 18:56:00.744758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.744718    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istiod-ca-cert" pod "ec4eaec1-65e4-4507-b4fd-7938a9adcd87" (UID: "ec4eaec1-65e4-4507-b4fd-7938a9adcd87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:56:00.745028 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.745004    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-certs" pod "ec4eaec1-65e4-4507-b4fd-7938a9adcd87" (UID: "ec4eaec1-65e4-4507-b4fd-7938a9adcd87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:00.745281 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.745256    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:00.745416 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.745285    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:00.745416 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.745301    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:00.745587 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.745512    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-socket" pod "ec4eaec1-65e4-4507-b4fd-7938a9adcd87" (UID: "ec4eaec1-65e4-4507-b4fd-7938a9adcd87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:00.745587 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.745558    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-data" pod "ec4eaec1-65e4-4507-b4fd-7938a9adcd87" (UID: "ec4eaec1-65e4-4507-b4fd-7938a9adcd87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:00.746286 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.746265    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-podinfo" pod "ec4eaec1-65e4-4507-b4fd-7938a9adcd87" (UID: "ec4eaec1-65e4-4507-b4fd-7938a9adcd87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:56:00.746286 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.746272    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-kube-api-access-pb4ws" pod "ec4eaec1-65e4-4507-b4fd-7938a9adcd87" (UID: "ec4eaec1-65e4-4507-b4fd-7938a9adcd87"). InnerVolumeSpecName "kube-api-access-pb4ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:00.746703 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.746685    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-token" pod "ec4eaec1-65e4-4507-b4fd-7938a9adcd87" (UID: "ec4eaec1-65e4-4507-b4fd-7938a9adcd87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:00.746784 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.746766    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-envoy" pod "ec4eaec1-65e4-4507-b4fd-7938a9adcd87" (UID: "ec4eaec1-65e4-4507-b4fd-7938a9adcd87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:00.846251 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.846172    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:00.846251 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.846201    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:00.846251 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.846211    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:00.846251 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.846219    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:00.846251 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.846228    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb4ws\" (UniqueName: \"kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-kube-api-access-pb4ws\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:00.846251 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:00.846238    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4eaec1-65e4-4507-b4fd-7938a9adcd87-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:01.271542 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:01.271509    4484 generic.go:356] "Generic (PLEG): container finished" podID="ec4eaec1-65e4-4507-b4fd-7938a9adcd87" containerID="e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c" exitCode=0
Oct 31 18:56:01.271696 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:01.271554    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"
Oct 31 18:56:01.271696 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:01.271561    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n" event={"ID":"ec4eaec1-65e4-4507-b4fd-7938a9adcd87","Type":"ContainerDied","Data":"e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c"}
Oct 31 18:56:01.271696 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:01.271602    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n" event={"ID":"ec4eaec1-65e4-4507-b4fd-7938a9adcd87","Type":"ContainerDied","Data":"c7547935f908653398c17424b7de5be7d6e6b40deba51b83aa9a79528d60ef04"}
Oct 31 18:56:01.271696 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:01.271622    4484 scope.go:122] "RemoveContainer" containerID="e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c"
Oct 31 18:56:01.282363 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:01.282346    4484 scope.go:122] "RemoveContainer" containerID="e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c"
Oct 31 18:56:01.282626 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:56:01.282607    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c\": container with ID starting with e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c not found: ID does not exist" containerID="e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c"
Oct 31 18:56:01.282683 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:01.282634    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c"} err="failed to get container status \"e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c\": rpc error: code = NotFound desc = could not find container \"e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c\": container with ID starting with e2ebdeef605b0083e7e9693698e78b81dbd3b7e6c4e716c55a451a9f7869931c not found: ID does not exist"
Oct 31 18:56:01.487850 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:01.487818    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"]
Oct 31 18:56:01.539675 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:01.539498    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-bkm4n"]
Oct 31 18:56:01.993745 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:01.993713    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4eaec1-65e4-4507-b4fd-7938a9adcd87" path="/var/lib/kubelet/pods/ec4eaec1-65e4-4507-b4fd-7938a9adcd87/volumes"
Oct 31 18:56:02.393649 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:02.393566    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:02.561784 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:02.561750    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:02.708623 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:02.708545    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:03.967921 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:03.967878    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc" podUID="79249a1b-9ab2-4969-95b5-51777c695a77" containerName="payload-processing" containerID="cri-o://d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f" gracePeriod=25
Oct 31 18:56:04.074563 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.074502    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps" podUID="1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71" containerName="payload-pre-processing" containerID="cri-o://8c2883b56f525874716366b23edc5916ff52f287a3b9587fd623a76c3620d2a4" gracePeriod=25
Oct 31 18:56:04.250581 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.250557    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"
Oct 31 18:56:04.307574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.307537    4484 generic.go:356] "Generic (PLEG): container finished" podID="1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71" containerID="8c2883b56f525874716366b23edc5916ff52f287a3b9587fd623a76c3620d2a4" exitCode=0
Oct 31 18:56:04.307757 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.307660    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps" event={"ID":"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71","Type":"ContainerDied","Data":"8c2883b56f525874716366b23edc5916ff52f287a3b9587fd623a76c3620d2a4"}
Oct 31 18:56:04.309296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.309270    4484 generic.go:356] "Generic (PLEG): container finished" podID="79249a1b-9ab2-4969-95b5-51777c695a77" containerID="d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f" exitCode=0
Oct 31 18:56:04.309420 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.309399    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"
Oct 31 18:56:04.309525 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.309418    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc" event={"ID":"79249a1b-9ab2-4969-95b5-51777c695a77","Type":"ContainerDied","Data":"d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f"}
Oct 31 18:56:04.309525 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.309501    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc" event={"ID":"79249a1b-9ab2-4969-95b5-51777c695a77","Type":"ContainerDied","Data":"608a846c9c81847417f30fbd668e534f63628f6f0f746ce5305cc2b37e28f7d0"}
Oct 31 18:56:04.309643 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.309536    4484 scope.go:122] "RemoveContainer" containerID="d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f"
Oct 31 18:56:04.319316 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.319297    4484 scope.go:122] "RemoveContainer" containerID="d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f"
Oct 31 18:56:04.319634 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:56:04.319599    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f\": container with ID starting with d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f not found: ID does not exist" containerID="d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f"
Oct 31 18:56:04.319699 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.319646    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f"} err="failed to get container status \"d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f\": rpc error: code = NotFound desc = could not find container \"d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f\": container with ID starting with d97cf7506f591ff2942a1c09f7b0444a624644dca19d2a6e6ca50bfa289caf1f not found: ID does not exist"
Oct 31 18:56:04.353782 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.353760    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"
Oct 31 18:56:04.395450 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.395421    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79249a1b-9ab2-4969-95b5-51777c695a77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79249a1b-9ab2-4969-95b5-51777c695a77-plugins-config-volume\") pod \"79249a1b-9ab2-4969-95b5-51777c695a77\" (UID: \"79249a1b-9ab2-4969-95b5-51777c695a77\") "
Oct 31 18:56:04.395626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.395551    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79249a1b-9ab2-4969-95b5-51777c695a77-kube-api-access-gjk4q\" (UniqueName: \"kubernetes.io/projected/79249a1b-9ab2-4969-95b5-51777c695a77-kube-api-access-gjk4q\") pod \"79249a1b-9ab2-4969-95b5-51777c695a77\" (UID: \"79249a1b-9ab2-4969-95b5-51777c695a77\") "
Oct 31 18:56:04.395868 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.395840    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79249a1b-9ab2-4969-95b5-51777c695a77-plugins-config-volume" pod "79249a1b-9ab2-4969-95b5-51777c695a77" (UID: "79249a1b-9ab2-4969-95b5-51777c695a77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:56:04.396002 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.395954    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79249a1b-9ab2-4969-95b5-51777c695a77-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:04.397790 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.397766    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79249a1b-9ab2-4969-95b5-51777c695a77-kube-api-access-gjk4q" pod "79249a1b-9ab2-4969-95b5-51777c695a77" (UID: "79249a1b-9ab2-4969-95b5-51777c695a77"). InnerVolumeSpecName "kube-api-access-gjk4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:04.497325 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.497289    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-plugins-config-volume\") pod \"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71\" (UID: \"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71\") "
Oct 31 18:56:04.497533 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.497350    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-kube-api-access-cvq6l\" (UniqueName: \"kubernetes.io/projected/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-kube-api-access-cvq6l\") pod \"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71\" (UID: \"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71\") "
Oct 31 18:56:04.497778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.497748    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-plugins-config-volume" pod "1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71" (UID: "1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:56:04.497937 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.497869    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:04.497937 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.497889    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjk4q\" (UniqueName: \"kubernetes.io/projected/79249a1b-9ab2-4969-95b5-51777c695a77-kube-api-access-gjk4q\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:04.499826 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.499804    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-kube-api-access-cvq6l" pod "1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71" (UID: "1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71"). InnerVolumeSpecName "kube-api-access-cvq6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:04.599231 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.599202    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvq6l\" (UniqueName: \"kubernetes.io/projected/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71-kube-api-access-cvq6l\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:04.730595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.730566    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"]
Oct 31 18:56:04.734659 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:04.734633    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e9c518-5f89c7bd99-n26fc"]
Oct 31 18:56:05.318268 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:05.318230    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps" event={"ID":"1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71","Type":"ContainerDied","Data":"35131ca1e423325babfb13c07d10c391499b8f9257225578548ac3554d7ec90e"}
Oct 31 18:56:05.318268 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:05.318269    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"
Oct 31 18:56:05.318845 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:05.318301    4484 scope.go:122] "RemoveContainer" containerID="8c2883b56f525874716366b23edc5916ff52f287a3b9587fd623a76c3620d2a4"
Oct 31 18:56:05.363984 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:05.363957    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:05.484463 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:05.484429    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"]
Oct 31 18:56:05.510895 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:05.510861    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e9c518-65885d4b67-vc2ps"]
Oct 31 18:56:05.802622 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:05.802587    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:05.994622 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:05.994589    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71" path="/var/lib/kubelet/pods/1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71/volumes"
Oct 31 18:56:05.995010 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:05.994997    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79249a1b-9ab2-4969-95b5-51777c695a77" path="/var/lib/kubelet/pods/79249a1b-9ab2-4969-95b5-51777c695a77/volumes"
Oct 31 18:56:07.245868 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:07.245835    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:11.898526 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:11.898494    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:16.560237 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:16.560198    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:20.502857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.502822    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"]
Oct 31 18:56:20.504094 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.503082    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k" podUID="ef8decda-dfc2-4801-89dc-8be430938bfa" containerName="maas-api" containerID="cri-o://addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729" gracePeriod=30
Oct 31 18:56:20.633574 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.633539    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:20.762460 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.762395    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:56:20.876153 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.876122    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-maas-api-tls\") pod \"ef8decda-dfc2-4801-89dc-8be430938bfa\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") "
Oct 31 18:56:20.876348 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.876200    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8decda-dfc2-4801-89dc-8be430938bfa-kube-api-access-8g9hw\" (UniqueName: \"kubernetes.io/projected/ef8decda-dfc2-4801-89dc-8be430938bfa-kube-api-access-8g9hw\") pod \"ef8decda-dfc2-4801-89dc-8be430938bfa\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") "
Oct 31 18:56:20.876348 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.876295    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef8decda-dfc2-4801-89dc-8be430938bfa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8decda-dfc2-4801-89dc-8be430938bfa-openshift-service-ca\") pod \"ef8decda-dfc2-4801-89dc-8be430938bfa\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") "
Oct 31 18:56:20.876454 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.876387    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-metrics-certs\") pod \"ef8decda-dfc2-4801-89dc-8be430938bfa\" (UID: \"ef8decda-dfc2-4801-89dc-8be430938bfa\") "
Oct 31 18:56:20.877052 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.877000    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8decda-dfc2-4801-89dc-8be430938bfa-openshift-service-ca" pod "ef8decda-dfc2-4801-89dc-8be430938bfa" (UID: "ef8decda-dfc2-4801-89dc-8be430938bfa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:56:20.881162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.881120    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-metrics-certs" pod "ef8decda-dfc2-4801-89dc-8be430938bfa" (UID: "ef8decda-dfc2-4801-89dc-8be430938bfa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:56:20.892932 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.892902    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8decda-dfc2-4801-89dc-8be430938bfa-kube-api-access-8g9hw" pod "ef8decda-dfc2-4801-89dc-8be430938bfa" (UID: "ef8decda-dfc2-4801-89dc-8be430938bfa"). InnerVolumeSpecName "kube-api-access-8g9hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:20.896726 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.896548    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-maas-api-tls" pod "ef8decda-dfc2-4801-89dc-8be430938bfa" (UID: "ef8decda-dfc2-4801-89dc-8be430938bfa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 18:56:20.952236 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.952205    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"]
Oct 31 18:56:20.952442 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.952414    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh" podUID="f96bd01d-ee97-4e78-8171-df82fe0513c5" containerName="payload-processing" containerID="cri-o://ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d" gracePeriod=30
Oct 31 18:56:20.977861 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.977839    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:20.977944 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.977863    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g9hw\" (UniqueName: \"kubernetes.io/projected/ef8decda-dfc2-4801-89dc-8be430938bfa-kube-api-access-8g9hw\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:20.977944 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.977875    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8decda-dfc2-4801-89dc-8be430938bfa-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:20.977944 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:20.977884    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef8decda-dfc2-4801-89dc-8be430938bfa-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:21.006087 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.006063    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"]
Oct 31 18:56:21.006245 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.006227    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv" podUID="1e18c6d0-893d-4a0c-a021-e8354d7be8e5" containerName="payload-pre-processing" containerID="cri-o://83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d" gracePeriod=30
Oct 31 18:56:21.125288 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.125258    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:21.259292 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.259267    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"
Oct 31 18:56:21.278394 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.278320    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:21.301285 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.301264    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"
Oct 31 18:56:21.392824 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.392799    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96bd01d-ee97-4e78-8171-df82fe0513c5-kube-api-access-l8l7r\" (UniqueName: \"kubernetes.io/projected/f96bd01d-ee97-4e78-8171-df82fe0513c5-kube-api-access-l8l7r\") pod \"f96bd01d-ee97-4e78-8171-df82fe0513c5\" (UID: \"f96bd01d-ee97-4e78-8171-df82fe0513c5\") "
Oct 31 18:56:21.393009 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.392919    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-plugins-config-volume\") pod \"1e18c6d0-893d-4a0c-a021-e8354d7be8e5\" (UID: \"1e18c6d0-893d-4a0c-a021-e8354d7be8e5\") "
Oct 31 18:56:21.393009 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.392956    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-kube-api-access-pnldv\" (UniqueName: \"kubernetes.io/projected/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-kube-api-access-pnldv\") pod \"1e18c6d0-893d-4a0c-a021-e8354d7be8e5\" (UID: \"1e18c6d0-893d-4a0c-a021-e8354d7be8e5\") "
Oct 31 18:56:21.393009 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.392977    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96bd01d-ee97-4e78-8171-df82fe0513c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96bd01d-ee97-4e78-8171-df82fe0513c5-plugins-config-volume\") pod \"f96bd01d-ee97-4e78-8171-df82fe0513c5\" (UID: \"f96bd01d-ee97-4e78-8171-df82fe0513c5\") "
Oct 31 18:56:21.393421 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.393388    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-plugins-config-volume" pod "1e18c6d0-893d-4a0c-a021-e8354d7be8e5" (UID: "1e18c6d0-893d-4a0c-a021-e8354d7be8e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:56:21.393421 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.393402    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96bd01d-ee97-4e78-8171-df82fe0513c5-plugins-config-volume" pod "f96bd01d-ee97-4e78-8171-df82fe0513c5" (UID: "f96bd01d-ee97-4e78-8171-df82fe0513c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:56:21.395069 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.395046    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96bd01d-ee97-4e78-8171-df82fe0513c5-kube-api-access-l8l7r" pod "f96bd01d-ee97-4e78-8171-df82fe0513c5" (UID: "f96bd01d-ee97-4e78-8171-df82fe0513c5"). InnerVolumeSpecName "kube-api-access-l8l7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:21.395529 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.395511    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-kube-api-access-pnldv" pod "1e18c6d0-893d-4a0c-a021-e8354d7be8e5" (UID: "1e18c6d0-893d-4a0c-a021-e8354d7be8e5"). InnerVolumeSpecName "kube-api-access-pnldv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:21.421040 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.421013    4484 generic.go:356] "Generic (PLEG): container finished" podID="1e18c6d0-893d-4a0c-a021-e8354d7be8e5" containerID="83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d" exitCode=0
Oct 31 18:56:21.421164 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.421110    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"
Oct 31 18:56:21.421216 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.421109    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv" event={"ID":"1e18c6d0-893d-4a0c-a021-e8354d7be8e5","Type":"ContainerDied","Data":"83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d"}
Oct 31 18:56:21.421263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.421249    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv" event={"ID":"1e18c6d0-893d-4a0c-a021-e8354d7be8e5","Type":"ContainerDied","Data":"ee841f440dfe6b61786a45a39aa23a47f199ed7b04425f7e714518aef1e10e9b"}
Oct 31 18:56:21.421316 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.421281    4484 scope.go:122] "RemoveContainer" containerID="83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d"
Oct 31 18:56:21.422585 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.422560    4484 generic.go:356] "Generic (PLEG): container finished" podID="f96bd01d-ee97-4e78-8171-df82fe0513c5" containerID="ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d" exitCode=0
Oct 31 18:56:21.422667 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.422590    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh" event={"ID":"f96bd01d-ee97-4e78-8171-df82fe0513c5","Type":"ContainerDied","Data":"ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d"}
Oct 31 18:56:21.422667 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.422605    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"
Oct 31 18:56:21.422667 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.422640    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh" event={"ID":"f96bd01d-ee97-4e78-8171-df82fe0513c5","Type":"ContainerDied","Data":"b587067d0a0b98cfa1e3c3f73da4c1f26ebc370f1ff2c6fd26aff77adb006f34"}
Oct 31 18:56:21.424101 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.424076    4484 generic.go:356] "Generic (PLEG): container finished" podID="ef8decda-dfc2-4801-89dc-8be430938bfa" containerID="addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729" exitCode=0
Oct 31 18:56:21.424217 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.424128    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"
Oct 31 18:56:21.424217 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.424163    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k" event={"ID":"ef8decda-dfc2-4801-89dc-8be430938bfa","Type":"ContainerDied","Data":"addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729"}
Oct 31 18:56:21.424217 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.424200    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k" event={"ID":"ef8decda-dfc2-4801-89dc-8be430938bfa","Type":"ContainerDied","Data":"67bf15a817b6b4b76ed154e65b672199f97345882f2c794c6535a3cf9f97ebe4"}
Oct 31 18:56:21.436115 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.436093    4484 scope.go:122] "RemoveContainer" containerID="83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d"
Oct 31 18:56:21.436466 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:56:21.436448    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d\": container with ID starting with 83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d not found: ID does not exist" containerID="83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d"
Oct 31 18:56:21.436567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.436506    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d"} err="failed to get container status \"83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d\": rpc error: code = NotFound desc = could not find container \"83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d\": container with ID starting with 83ab4f4f119731b5d00585319d2800e039169790a57c1410c3965294ad942d4d not found: ID does not exist"
Oct 31 18:56:21.436567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.436525    4484 scope.go:122] "RemoveContainer" containerID="ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d"
Oct 31 18:56:21.445202 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.445185    4484 scope.go:122] "RemoveContainer" containerID="ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d"
Oct 31 18:56:21.445446 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:56:21.445428    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d\": container with ID starting with ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d not found: ID does not exist" containerID="ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d"
Oct 31 18:56:21.445518 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.445452    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d"} err="failed to get container status \"ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d\": rpc error: code = NotFound desc = could not find container \"ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d\": container with ID starting with ad643eac8ccb118fe00c2b21ae6c98c0f5bc46026842ef85e52ded27e9ae547d not found: ID does not exist"
Oct 31 18:56:21.445518 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.445487    4484 scope.go:122] "RemoveContainer" containerID="addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729"
Oct 31 18:56:21.453597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.453578    4484 scope.go:122] "RemoveContainer" containerID="addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729"
Oct 31 18:56:21.453849 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:56:21.453832    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729\": container with ID starting with addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729 not found: ID does not exist" containerID="addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729"
Oct 31 18:56:21.453918 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.453854    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729"} err="failed to get container status \"addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729\": rpc error: code = NotFound desc = could not find container \"addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729\": container with ID starting with addbf02af6d89f55910e2a0053366b3c6d5beb60f2d8caa8f2383363f9731729 not found: ID does not exist"
Oct 31 18:56:21.521495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.509252    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8l7r\" (UniqueName: \"kubernetes.io/projected/f96bd01d-ee97-4e78-8171-df82fe0513c5-kube-api-access-l8l7r\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:21.521495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.509278    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:21.521495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.509289    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnldv\" (UniqueName: \"kubernetes.io/projected/1e18c6d0-893d-4a0c-a021-e8354d7be8e5-kube-api-access-pnldv\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:21.521495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.509299    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96bd01d-ee97-4e78-8171-df82fe0513c5-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:21.631099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.631067    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"]
Oct 31 18:56:21.638941 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.638894    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-335e78-6f7c959986-tmwlh"]
Oct 31 18:56:21.747516 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.747467    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"]
Oct 31 18:56:21.753113 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.753088    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-335e78-6fc67bcb87-wnpwv"]
Oct 31 18:56:21.863577 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.863463    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"]
Oct 31 18:56:21.867279 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.867260    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-335e78-89bc5c4fc-k2x4k"]
Oct 31 18:56:21.993464 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.993432    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e18c6d0-893d-4a0c-a021-e8354d7be8e5" path="/var/lib/kubelet/pods/1e18c6d0-893d-4a0c-a021-e8354d7be8e5/volumes"
Oct 31 18:56:21.993933 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.993916    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8decda-dfc2-4801-89dc-8be430938bfa" path="/var/lib/kubelet/pods/ef8decda-dfc2-4801-89dc-8be430938bfa/volumes"
Oct 31 18:56:21.994269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:21.994257    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96bd01d-ee97-4e78-8171-df82fe0513c5" path="/var/lib/kubelet/pods/f96bd01d-ee97-4e78-8171-df82fe0513c5/volumes"
Oct 31 18:56:22.534457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:22.534421    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:25.030189 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:25.030149    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:25.198717 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:25.198683    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:25.474335 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:56:25.474249    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:56:25.479113 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:25.479079    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:26.559962 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:26.559919    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:26.720286 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:26.720243    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:26.881948 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:26.881860    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:27.037138 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.037108    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"]
Oct 31 18:56:27.037342 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.037320    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy" containerID="cri-o://7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5" gracePeriod=30
Oct 31 18:56:27.060052 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.060021    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:27.279071 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.279038    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"]
Oct 31 18:56:27.279281 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.279243    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm" podUID="05c4f0a0-e452-4fb3-b72c-5dda7639c882" containerName="payload-processing" containerID="cri-o://1a1d30a3674b61c59d3d888a890b2e41055f77d770b33ba36ee079e85485b56e" gracePeriod=30
Oct 31 18:56:27.373769 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.373737    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"]
Oct 31 18:56:27.374275 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.374020    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59" podUID="c56988bf-4c50-48b4-b276-b3501f735545" containerName="payload-pre-processing" containerID="cri-o://41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489" gracePeriod=30
Oct 31 18:56:27.400051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.400021    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:27.470281 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.470256    4484 generic.go:356] "Generic (PLEG): container finished" podID="05c4f0a0-e452-4fb3-b72c-5dda7639c882" containerID="1a1d30a3674b61c59d3d888a890b2e41055f77d770b33ba36ee079e85485b56e" exitCode=0
Oct 31 18:56:27.470449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.470337    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm" event={"ID":"05c4f0a0-e452-4fb3-b72c-5dda7639c882","Type":"ContainerDied","Data":"1a1d30a3674b61c59d3d888a890b2e41055f77d770b33ba36ee079e85485b56e"}
Oct 31 18:56:27.551295 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.551270    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"
Oct 31 18:56:27.559127 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.559042    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:27.656393 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.656367    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"
Oct 31 18:56:27.678695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.678666    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c4f0a0-e452-4fb3-b72c-5dda7639c882-kube-api-access-bwx6r\" (UniqueName: \"kubernetes.io/projected/05c4f0a0-e452-4fb3-b72c-5dda7639c882-kube-api-access-bwx6r\") pod \"05c4f0a0-e452-4fb3-b72c-5dda7639c882\" (UID: \"05c4f0a0-e452-4fb3-b72c-5dda7639c882\") "
Oct 31 18:56:27.678925 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.678907    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c4f0a0-e452-4fb3-b72c-5dda7639c882-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c4f0a0-e452-4fb3-b72c-5dda7639c882-plugins-config-volume\") pod \"05c4f0a0-e452-4fb3-b72c-5dda7639c882\" (UID: \"05c4f0a0-e452-4fb3-b72c-5dda7639c882\") "
Oct 31 18:56:27.679269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.679248    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c4f0a0-e452-4fb3-b72c-5dda7639c882-plugins-config-volume" pod "05c4f0a0-e452-4fb3-b72c-5dda7639c882" (UID: "05c4f0a0-e452-4fb3-b72c-5dda7639c882"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:56:27.679427 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.679412    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c4f0a0-e452-4fb3-b72c-5dda7639c882-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:27.680855 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.680832    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c4f0a0-e452-4fb3-b72c-5dda7639c882-kube-api-access-bwx6r" pod "05c4f0a0-e452-4fb3-b72c-5dda7639c882" (UID: "05c4f0a0-e452-4fb3-b72c-5dda7639c882"). InnerVolumeSpecName "kube-api-access-bwx6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:27.780235 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.780195    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56988bf-4c50-48b4-b276-b3501f735545-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56988bf-4c50-48b4-b276-b3501f735545-plugins-config-volume\") pod \"c56988bf-4c50-48b4-b276-b3501f735545\" (UID: \"c56988bf-4c50-48b4-b276-b3501f735545\") "
Oct 31 18:56:27.780436 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.780265    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56988bf-4c50-48b4-b276-b3501f735545-kube-api-access-f2cbp\" (UniqueName: \"kubernetes.io/projected/c56988bf-4c50-48b4-b276-b3501f735545-kube-api-access-f2cbp\") pod \"c56988bf-4c50-48b4-b276-b3501f735545\" (UID: \"c56988bf-4c50-48b4-b276-b3501f735545\") "
Oct 31 18:56:27.780743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.780674    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56988bf-4c50-48b4-b276-b3501f735545-plugins-config-volume" pod "c56988bf-4c50-48b4-b276-b3501f735545" (UID: "c56988bf-4c50-48b4-b276-b3501f735545"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:56:27.780912 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.780891    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwx6r\" (UniqueName: \"kubernetes.io/projected/05c4f0a0-e452-4fb3-b72c-5dda7639c882-kube-api-access-bwx6r\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:27.780986 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.780918    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56988bf-4c50-48b4-b276-b3501f735545-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:27.783090 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.783065    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56988bf-4c50-48b4-b276-b3501f735545-kube-api-access-f2cbp" pod "c56988bf-4c50-48b4-b276-b3501f735545" (UID: "c56988bf-4c50-48b4-b276-b3501f735545"). InnerVolumeSpecName "kube-api-access-f2cbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:27.882067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:27.882032    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2cbp\" (UniqueName: \"kubernetes.io/projected/c56988bf-4c50-48b4-b276-b3501f735545-kube-api-access-f2cbp\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:28.332098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.332064    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 18:56:28.332237 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.332121    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 18:56:28.477457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.477428    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"
Oct 31 18:56:28.477633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.477490    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm" event={"ID":"05c4f0a0-e452-4fb3-b72c-5dda7639c882","Type":"ContainerDied","Data":"b9e4beb2a7727e4df34a1c1aaa1f096cb78ac3cab4bc60f13cfdfaf057fd8955"}
Oct 31 18:56:28.477633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.477558    4484 scope.go:122] "RemoveContainer" containerID="1a1d30a3674b61c59d3d888a890b2e41055f77d770b33ba36ee079e85485b56e"
Oct 31 18:56:28.478910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.478874    4484 generic.go:356] "Generic (PLEG): container finished" podID="c56988bf-4c50-48b4-b276-b3501f735545" containerID="41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489" exitCode=0
Oct 31 18:56:28.478982 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.478913    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59" event={"ID":"c56988bf-4c50-48b4-b276-b3501f735545","Type":"ContainerDied","Data":"41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489"}
Oct 31 18:56:28.478982 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.478924    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"
Oct 31 18:56:28.478982 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.478944    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59" event={"ID":"c56988bf-4c50-48b4-b276-b3501f735545","Type":"ContainerDied","Data":"3fc6dcbb20e7a84239c64f9442b0ea2acc8320383898f42be0876f4b69780bb1"}
Oct 31 18:56:28.487100 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.487077    4484 scope.go:122] "RemoveContainer" containerID="41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489"
Oct 31 18:56:28.495905 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.495888    4484 scope.go:122] "RemoveContainer" containerID="41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489"
Oct 31 18:56:28.496188 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:56:28.496166    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489\": container with ID starting with 41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489 not found: ID does not exist" containerID="41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489"
Oct 31 18:56:28.496288 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.496195    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489"} err="failed to get container status \"41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489\": rpc error: code = NotFound desc = could not find container \"41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489\": container with ID starting with 41f0d287d79d89a4f3664df47235180693ed6b34234d6fed2a3581444906f489 not found: ID does not exist"
Oct 31 18:56:28.581567 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.581461    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:28.643969 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.643942    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"]
Oct 31 18:56:28.694368 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.694340    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ceef3b-5c4dc7bcd7-nnr59"]
Oct 31 18:56:28.758350 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.758318    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"]
Oct 31 18:56:28.763017 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.762993    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ceef3b-57d5cfcc5b-rgctm"]
Oct 31 18:56:28.801092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:28.801068    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:29.471363 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:29.471319    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:29.719533 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:29.719500    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:29.993326 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:29.993289    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c4f0a0-e452-4fb3-b72c-5dda7639c882" path="/var/lib/kubelet/pods/05c4f0a0-e452-4fb3-b72c-5dda7639c882/volumes"
Oct 31 18:56:29.993733 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:29.993719    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56988bf-4c50-48b4-b276-b3501f735545" path="/var/lib/kubelet/pods/c56988bf-4c50-48b4-b276-b3501f735545/volumes"
Oct 31 18:56:31.502976 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:31.502929    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:31.699767 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:31.699727    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:31.858392 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:31.858305    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:32.110009 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.109933    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:56:32.298722 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.298697    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:56:32.433407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.433330    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-kube-api-access-tk2vm\" (UniqueName: \"kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-kube-api-access-tk2vm\") pod \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") "
Oct 31 18:56:32.433407 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.433369    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istiod-ca-cert\") pod \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") "
Oct 31 18:56:32.433626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.433421    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-credential-socket\") pod \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") "
Oct 31 18:56:32.433626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.433453    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-data\") pod \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") "
Oct 31 18:56:32.433626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.433542    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-podinfo\") pod \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") "
Oct 31 18:56:32.433626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.433603    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-token\" (UniqueName: \"kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-token\") pod \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") "
Oct 31 18:56:32.433834 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.433796    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-credential-socket" pod "c5d585fa-0b3c-4049-a004-c5fab33a5f2c" (UID: "c5d585fa-0b3c-4049-a004-c5fab33a5f2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:32.433892 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.433634    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-certs\") pod \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") "
Oct 31 18:56:32.433892 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.433879    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istiod-ca-cert" pod "c5d585fa-0b3c-4049-a004-c5fab33a5f2c" (UID: "c5d585fa-0b3c-4049-a004-c5fab33a5f2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:56:32.434006 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.433896    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-envoy\") pod \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") "
Oct 31 18:56:32.434006 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.433940    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-socket\") pod \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\" (UID: \"c5d585fa-0b3c-4049-a004-c5fab33a5f2c\") "
Oct 31 18:56:32.434136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.434090    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-certs" pod "c5d585fa-0b3c-4049-a004-c5fab33a5f2c" (UID: "c5d585fa-0b3c-4049-a004-c5fab33a5f2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:32.434221 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.434205    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-data" pod "c5d585fa-0b3c-4049-a004-c5fab33a5f2c" (UID: "c5d585fa-0b3c-4049-a004-c5fab33a5f2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:32.434359 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.434340    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-socket" pod "c5d585fa-0b3c-4049-a004-c5fab33a5f2c" (UID: "c5d585fa-0b3c-4049-a004-c5fab33a5f2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:32.434576 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.434552    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:32.434693 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.434578    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:32.434693 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.434593    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:32.434693 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.434631    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:32.434693 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.434644    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:32.435974 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.435950    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-kube-api-access-tk2vm" pod "c5d585fa-0b3c-4049-a004-c5fab33a5f2c" (UID: "c5d585fa-0b3c-4049-a004-c5fab33a5f2c"). InnerVolumeSpecName "kube-api-access-tk2vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:32.436387 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.436365    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-envoy" pod "c5d585fa-0b3c-4049-a004-c5fab33a5f2c" (UID: "c5d585fa-0b3c-4049-a004-c5fab33a5f2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 18:56:32.436677 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.436656    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-token" pod "c5d585fa-0b3c-4049-a004-c5fab33a5f2c" (UID: "c5d585fa-0b3c-4049-a004-c5fab33a5f2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:56:32.436771 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.436677    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-podinfo" pod "c5d585fa-0b3c-4049-a004-c5fab33a5f2c" (UID: "c5d585fa-0b3c-4049-a004-c5fab33a5f2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 18:56:32.506297 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.506268    4484 generic.go:356] "Generic (PLEG): container finished" podID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerID="7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5" exitCode=0
Oct 31 18:56:32.506778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.506318    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"
Oct 31 18:56:32.506778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.506359    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" event={"ID":"c5d585fa-0b3c-4049-a004-c5fab33a5f2c","Type":"ContainerDied","Data":"7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5"}
Oct 31 18:56:32.506778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.506421    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4" event={"ID":"c5d585fa-0b3c-4049-a004-c5fab33a5f2c","Type":"ContainerDied","Data":"3a62192cd0f945d551ec35388b15f8af3dd6ab562bfe28631a2421865b8afdd2"}
Oct 31 18:56:32.506778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.506457    4484 scope.go:122] "RemoveContainer" containerID="7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5"
Oct 31 18:56:32.516745 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.516728    4484 scope.go:122] "RemoveContainer" containerID="1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4"
Oct 31 18:56:32.525196 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.525181    4484 scope.go:122] "RemoveContainer" containerID="7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5"
Oct 31 18:56:32.525486 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:56:32.525456    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5\": container with ID starting with 7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5 not found: ID does not exist" containerID="7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5"
Oct 31 18:56:32.525570 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.525510    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5"} err="failed to get container status \"7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5\": rpc error: code = NotFound desc = could not find container \"7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5\": container with ID starting with 7428feeb63c98c9a97b865a71084a48ca14c979e58959f5a4b875b3dc1276ef5 not found: ID does not exist"
Oct 31 18:56:32.525570 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.525529    4484 scope.go:122] "RemoveContainer" containerID="1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4"
Oct 31 18:56:32.525777 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:56:32.525759    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4\": container with ID starting with 1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4 not found: ID does not exist" containerID="1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4"
Oct 31 18:56:32.525826 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.525785    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4"} err="failed to get container status \"1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4\": rpc error: code = NotFound desc = could not find container \"1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4\": container with ID starting with 1573ff0262da8bd85f073e519624cebe1362822a266e9172633f1e17a19a57e4 not found: ID does not exist"
Oct 31 18:56:32.535435 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.535418    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:32.535523 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.535437    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:32.535523 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.535448    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:32.535523 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.535456    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk2vm\" (UniqueName: \"kubernetes.io/projected/c5d585fa-0b3c-4049-a004-c5fab33a5f2c-kube-api-access-tk2vm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:56:32.675788 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.675749    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"]
Oct 31 18:56:32.678145 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:32.678122    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-335e78-openshift-default-6d6884f755-bchd4"]
Oct 31 18:56:33.588934 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:33.588892    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:33.795110 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:33.795071    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:33.994301 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:33.994272    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" path="/var/lib/kubelet/pods/c5d585fa-0b3c-4049-a004-c5fab33a5f2c/volumes"
Oct 31 18:56:34.140330 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:34.140296    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:34.710259 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:34.710217    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:34.873291 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:34.873258    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:35.089242 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089213    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"]
Oct 31 18:56:35.089746 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089733    4484 cpu_manager.go:414] "containerMap: removing container" podUID="1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71" containerName="payload-pre-processing"
Oct 31 18:56:35.089806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089748    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71" containerName="payload-pre-processing"
Oct 31 18:56:35.089806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089766    4484 cpu_manager.go:414] "containerMap: removing container" podUID="74824d04-008e-4e17-8ff6-1060a8a990a2" containerName="payload-processing"
Oct 31 18:56:35.089806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089771    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74824d04-008e-4e17-8ff6-1060a8a990a2" containerName="payload-processing"
Oct 31 18:56:35.089806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089779    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ef8decda-dfc2-4801-89dc-8be430938bfa" containerName="maas-api"
Oct 31 18:56:35.089806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089783    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8decda-dfc2-4801-89dc-8be430938bfa" containerName="maas-api"
Oct 31 18:56:35.089806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089795    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f96bd01d-ee97-4e78-8171-df82fe0513c5" containerName="payload-processing"
Oct 31 18:56:35.089806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089799    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96bd01d-ee97-4e78-8171-df82fe0513c5" containerName="payload-processing"
Oct 31 18:56:35.089806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089806    4484 cpu_manager.go:414] "containerMap: removing container" podUID="79249a1b-9ab2-4969-95b5-51777c695a77" containerName="payload-processing"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089811    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79249a1b-9ab2-4969-95b5-51777c695a77" containerName="payload-processing"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089829    4484 cpu_manager.go:414] "containerMap: removing container" podUID="1e18c6d0-893d-4a0c-a021-e8354d7be8e5" containerName="payload-pre-processing"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089836    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e18c6d0-893d-4a0c-a021-e8354d7be8e5" containerName="payload-pre-processing"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089849    4484 cpu_manager.go:414] "containerMap: removing container" podUID="711334f9-d214-4503-bf64-4e0b8bc408a6" containerName="payload-processing"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089856    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711334f9-d214-4503-bf64-4e0b8bc408a6" containerName="payload-processing"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089872    4484 cpu_manager.go:414] "containerMap: removing container" podUID="24f9a5e2-42a9-4f4a-9289-8bb7699eb450" containerName="maas-api"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089879    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f9a5e2-42a9-4f4a-9289-8bb7699eb450" containerName="maas-api"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089889    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089896    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089913    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c56988bf-4c50-48b4-b276-b3501f735545" containerName="payload-pre-processing"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089918    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56988bf-4c50-48b4-b276-b3501f735545" containerName="payload-pre-processing"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089927    4484 cpu_manager.go:414] "containerMap: removing container" podUID="05c4f0a0-e452-4fb3-b72c-5dda7639c882" containerName="payload-processing"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089931    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c4f0a0-e452-4fb3-b72c-5dda7639c882" containerName="payload-processing"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089939    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ec4eaec1-65e4-4507-b4fd-7938a9adcd87" containerName="istio-proxy"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089943    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4eaec1-65e4-4507-b4fd-7938a9adcd87" containerName="istio-proxy"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089952    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.089956    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090038    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="74824d04-008e-4e17-8ff6-1060a8a990a2" containerName="payload-processing"
Oct 31 18:56:35.090039 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090045    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8decda-dfc2-4801-89dc-8be430938bfa" containerName="maas-api"
Oct 31 18:56:35.090593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090054    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96bd01d-ee97-4e78-8171-df82fe0513c5" containerName="payload-processing"
Oct 31 18:56:35.090593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090060    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="79249a1b-9ab2-4969-95b5-51777c695a77" containerName="payload-processing"
Oct 31 18:56:35.090593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090066    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e18c6d0-893d-4a0c-a021-e8354d7be8e5" containerName="payload-pre-processing"
Oct 31 18:56:35.090593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090071    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="711334f9-d214-4503-bf64-4e0b8bc408a6" containerName="payload-processing"
Oct 31 18:56:35.090593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090078    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f9a5e2-42a9-4f4a-9289-8bb7699eb450" containerName="maas-api"
Oct 31 18:56:35.090593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090083    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy"
Oct 31 18:56:35.090593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090089    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56988bf-4c50-48b4-b276-b3501f735545" containerName="payload-pre-processing"
Oct 31 18:56:35.090593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090095    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c4f0a0-e452-4fb3-b72c-5dda7639c882" containerName="payload-processing"
Oct 31 18:56:35.090593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090100    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4eaec1-65e4-4507-b4fd-7938a9adcd87" containerName="istio-proxy"
Oct 31 18:56:35.090593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090105    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d585fa-0b3c-4049-a004-c5fab33a5f2c" containerName="istio-proxy"
Oct 31 18:56:35.090593 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.090110    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3f0f92-f8e1-41f2-a5e6-5911bfff0c71" containerName="payload-pre-processing"
Oct 31 18:56:35.094900 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.094874    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"
Oct 31 18:56:35.150053 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.150028    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"]
Oct 31 18:56:35.160677 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.160652    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659c1e71-ce7c-4d01-8340-2263f57438ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh\" (UID: \"659c1e71-ce7c-4d01-8340-2263f57438ea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"
Oct 31 18:56:35.160814 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.160785    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khk8v\" (UniqueName: \"kubernetes.io/projected/659c1e71-ce7c-4d01-8340-2263f57438ea-kube-api-access-khk8v\") pod \"payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh\" (UID: \"659c1e71-ce7c-4d01-8340-2263f57438ea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"
Oct 31 18:56:35.198636 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.198603    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-35566a-dockercfg-d2dtc\""
Oct 31 18:56:35.198832 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.198719    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-35566a\""
Oct 31 18:56:35.254382 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.254351    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"]
Oct 31 18:56:35.258272 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.258251    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"
Oct 31 18:56:35.261867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.261846    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrggv\" (UniqueName: \"kubernetes.io/projected/d4904ff6-8feb-4491-9503-52734d32c17c-kube-api-access-qrggv\") pod \"payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q\" (UID: \"d4904ff6-8feb-4491-9503-52734d32c17c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"
Oct 31 18:56:35.261979 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.261903    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khk8v\" (UniqueName: \"kubernetes.io/projected/659c1e71-ce7c-4d01-8340-2263f57438ea-kube-api-access-khk8v\") pod \"payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh\" (UID: \"659c1e71-ce7c-4d01-8340-2263f57438ea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"
Oct 31 18:56:35.262092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.262075    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4904ff6-8feb-4491-9503-52734d32c17c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q\" (UID: \"d4904ff6-8feb-4491-9503-52734d32c17c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"
Oct 31 18:56:35.262165 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.262146    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659c1e71-ce7c-4d01-8340-2263f57438ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh\" (UID: \"659c1e71-ce7c-4d01-8340-2263f57438ea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"
Oct 31 18:56:35.262795 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.262779    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659c1e71-ce7c-4d01-8340-2263f57438ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh\" (UID: \"659c1e71-ce7c-4d01-8340-2263f57438ea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"
Oct 31 18:56:35.314572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.314543    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"]
Oct 31 18:56:35.322697 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.322668    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khk8v\" (UniqueName: \"kubernetes.io/projected/659c1e71-ce7c-4d01-8340-2263f57438ea-kube-api-access-khk8v\") pod \"payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh\" (UID: \"659c1e71-ce7c-4d01-8340-2263f57438ea\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"
Oct 31 18:56:35.363355 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.363290    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4904ff6-8feb-4491-9503-52734d32c17c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q\" (UID: \"d4904ff6-8feb-4491-9503-52734d32c17c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"
Oct 31 18:56:35.363535 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.363369    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrggv\" (UniqueName: \"kubernetes.io/projected/d4904ff6-8feb-4491-9503-52734d32c17c-kube-api-access-qrggv\") pod \"payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q\" (UID: \"d4904ff6-8feb-4491-9503-52734d32c17c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"
Oct 31 18:56:35.364137 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.364112    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4904ff6-8feb-4491-9503-52734d32c17c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q\" (UID: \"d4904ff6-8feb-4491-9503-52734d32c17c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"
Oct 31 18:56:35.408232 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.408206    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"
Oct 31 18:56:35.421021 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.420998    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrggv\" (UniqueName: \"kubernetes.io/projected/d4904ff6-8feb-4491-9503-52734d32c17c-kube-api-access-qrggv\") pod \"payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q\" (UID: \"d4904ff6-8feb-4491-9503-52734d32c17c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"
Oct 31 18:56:35.571250 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.571214    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"
Oct 31 18:56:35.781778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.781756    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"]
Oct 31 18:56:35.782860 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:56:35.782836    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659c1e71_ce7c_4d01_8340_2263f57438ea.slice/crio-e18872d112bbc76362240842301946af9d59426bdae9cbee99b2436ce3db87ba WatchSource:0}: Error finding container e18872d112bbc76362240842301946af9d59426bdae9cbee99b2436ce3db87ba: Status 404 returned error can't find the container with id e18872d112bbc76362240842301946af9d59426bdae9cbee99b2436ce3db87ba
Oct 31 18:56:35.857669 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.857641    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:35.943129 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:35.943105    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"]
Oct 31 18:56:35.943998 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:56:35.943971    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4904ff6_8feb_4491_9503_52734d32c17c.slice/crio-5408f107ac9dcfd931918525094f83c5daacb4cad8983b9e19d3fd4071c86df9 WatchSource:0}: Error finding container 5408f107ac9dcfd931918525094f83c5daacb4cad8983b9e19d3fd4071c86df9: Status 404 returned error can't find the container with id 5408f107ac9dcfd931918525094f83c5daacb4cad8983b9e19d3fd4071c86df9
Oct 31 18:56:36.329965 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:36.329931    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:36.478269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:36.478234    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:36.533653 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:36.533619    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh" event={"ID":"659c1e71-ce7c-4d01-8340-2263f57438ea","Type":"ContainerStarted","Data":"bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0"}
Oct 31 18:56:36.533828 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:36.533686    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"
Oct 31 18:56:36.533828 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:36.533709    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh" event={"ID":"659c1e71-ce7c-4d01-8340-2263f57438ea","Type":"ContainerStarted","Data":"e18872d112bbc76362240842301946af9d59426bdae9cbee99b2436ce3db87ba"}
Oct 31 18:56:36.535270 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:36.535245    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q" event={"ID":"d4904ff6-8feb-4491-9503-52734d32c17c","Type":"ContainerStarted","Data":"86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d"}
Oct 31 18:56:36.535349 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:36.535292    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q" event={"ID":"d4904ff6-8feb-4491-9503-52734d32c17c","Type":"ContainerStarted","Data":"5408f107ac9dcfd931918525094f83c5daacb4cad8983b9e19d3fd4071c86df9"}
Oct 31 18:56:36.535391 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:36.535365    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"
Oct 31 18:56:36.642122 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:36.642036    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:36.648906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:36.648861    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh" podStartSLOduration=1.6488494299999998 podStartE2EDuration="1.64884943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:56:36.648173679 +0000 UTC m=+2142.794623543" watchObservedRunningTime="2026-10-31 18:56:36.64884943 +0000 UTC m=+2142.795299294"
Oct 31 18:56:36.771291 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:36.771235    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q" podStartSLOduration=1.771219404 podStartE2EDuration="1.771219404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:56:36.766433454 +0000 UTC m=+2142.912883318" watchObservedRunningTime="2026-10-31 18:56:36.771219404 +0000 UTC m=+2142.917669320"
Oct 31 18:56:39.011626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:39.011595    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:39.216413 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:39.216375    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:39.873040 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:39.872998    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:40.038078 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.037923    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:40.191301 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.191206    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:40.307357 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.307325    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"]
Oct 31 18:56:40.312073 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.312049    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"
Oct 31 18:56:40.366092 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.366067    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"]
Oct 31 18:56:40.414055 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.414026    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4157da\""
Oct 31 18:56:40.414312 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.414149    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4157da-dockercfg-68ngb\""
Oct 31 18:56:40.423778 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.423750    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460a6553-ba00-4794-b626-79da1137b729-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg\" (UID: \"460a6553-ba00-4794-b626-79da1137b729\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"
Oct 31 18:56:40.423954 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.423935    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7plc\" (UniqueName: \"kubernetes.io/projected/460a6553-ba00-4794-b626-79da1137b729-kube-api-access-c7plc\") pod \"payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg\" (UID: \"460a6553-ba00-4794-b626-79da1137b729\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"
Oct 31 18:56:40.467098 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.467025    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"]
Oct 31 18:56:40.471047 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.471030    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"
Oct 31 18:56:40.524269 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.524234    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"]
Oct 31 18:56:40.524786 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.524757    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460a6553-ba00-4794-b626-79da1137b729-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg\" (UID: \"460a6553-ba00-4794-b626-79da1137b729\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"
Oct 31 18:56:40.524925 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.524880    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqndb\" (UniqueName: \"kubernetes.io/projected/9c9e4a95-1f7c-423b-b6ff-1715f739359b-kube-api-access-qqndb\") pod \"payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh\" (UID: \"9c9e4a95-1f7c-423b-b6ff-1715f739359b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"
Oct 31 18:56:40.524998 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.524941    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9e4a95-1f7c-423b-b6ff-1715f739359b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh\" (UID: \"9c9e4a95-1f7c-423b-b6ff-1715f739359b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"
Oct 31 18:56:40.525106 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.525082    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7plc\" (UniqueName: \"kubernetes.io/projected/460a6553-ba00-4794-b626-79da1137b729-kube-api-access-c7plc\") pod \"payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg\" (UID: \"460a6553-ba00-4794-b626-79da1137b729\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"
Oct 31 18:56:40.525540 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.525524    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460a6553-ba00-4794-b626-79da1137b729-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg\" (UID: \"460a6553-ba00-4794-b626-79da1137b729\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"
Oct 31 18:56:40.582404 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.582379    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7plc\" (UniqueName: \"kubernetes.io/projected/460a6553-ba00-4794-b626-79da1137b729-kube-api-access-c7plc\") pod \"payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg\" (UID: \"460a6553-ba00-4794-b626-79da1137b729\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"
Oct 31 18:56:40.625919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.625891    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqndb\" (UniqueName: \"kubernetes.io/projected/9c9e4a95-1f7c-423b-b6ff-1715f739359b-kube-api-access-qqndb\") pod \"payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh\" (UID: \"9c9e4a95-1f7c-423b-b6ff-1715f739359b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"
Oct 31 18:56:40.626083 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.625940    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9e4a95-1f7c-423b-b6ff-1715f739359b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh\" (UID: \"9c9e4a95-1f7c-423b-b6ff-1715f739359b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"
Oct 31 18:56:40.626135 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.626120    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"
Oct 31 18:56:40.626676 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.626658    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9e4a95-1f7c-423b-b6ff-1715f739359b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh\" (UID: \"9c9e4a95-1f7c-423b-b6ff-1715f739359b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"
Oct 31 18:56:40.686584 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.685871    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqndb\" (UniqueName: \"kubernetes.io/projected/9c9e4a95-1f7c-423b-b6ff-1715f739359b-kube-api-access-qqndb\") pod \"payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh\" (UID: \"9c9e4a95-1f7c-423b-b6ff-1715f739359b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"
Oct 31 18:56:40.784927 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.784897    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"
Oct 31 18:56:40.995342 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.995308    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:40.997991 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:40.997968    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"]
Oct 31 18:56:40.998115 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:56:40.998088    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460a6553_ba00_4794_b626_79da1137b729.slice/crio-a9247bfc9cf7b5e90226773370158af1efd4d6fa1619fbee6587392d51978a61 WatchSource:0}: Error finding container a9247bfc9cf7b5e90226773370158af1efd4d6fa1619fbee6587392d51978a61: Status 404 returned error can't find the container with id a9247bfc9cf7b5e90226773370158af1efd4d6fa1619fbee6587392d51978a61
Oct 31 18:56:41.158173 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:41.158150    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"]
Oct 31 18:56:41.159530 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:56:41.159503    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9e4a95_1f7c_423b_b6ff_1715f739359b.slice/crio-7477c02b9eaf3268e611d67ac696f7771e2a3313855a87b26b911a4bf1fbcf97 WatchSource:0}: Error finding container 7477c02b9eaf3268e611d67ac696f7771e2a3313855a87b26b911a4bf1fbcf97: Status 404 returned error can't find the container with id 7477c02b9eaf3268e611d67ac696f7771e2a3313855a87b26b911a4bf1fbcf97
Oct 31 18:56:41.565682 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:41.565646    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh" event={"ID":"9c9e4a95-1f7c-423b-b6ff-1715f739359b","Type":"ContainerStarted","Data":"2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6"}
Oct 31 18:56:41.565907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:41.565720    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh" event={"ID":"9c9e4a95-1f7c-423b-b6ff-1715f739359b","Type":"ContainerStarted","Data":"7477c02b9eaf3268e611d67ac696f7771e2a3313855a87b26b911a4bf1fbcf97"}
Oct 31 18:56:41.565907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:41.565818    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"
Oct 31 18:56:41.567455 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:41.567428    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg" event={"ID":"460a6553-ba00-4794-b626-79da1137b729","Type":"ContainerStarted","Data":"066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0"}
Oct 31 18:56:41.567596 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:41.567468    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg" event={"ID":"460a6553-ba00-4794-b626-79da1137b729","Type":"ContainerStarted","Data":"a9247bfc9cf7b5e90226773370158af1efd4d6fa1619fbee6587392d51978a61"}
Oct 31 18:56:41.567596 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:41.567510    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"
Oct 31 18:56:41.680645 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:41.680596    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh" podStartSLOduration=1.6805832870000001 podStartE2EDuration="1.680583287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:56:41.679321198 +0000 UTC m=+2147.825771062" watchObservedRunningTime="2026-10-31 18:56:41.680583287 +0000 UTC m=+2147.827033149"
Oct 31 18:56:41.793810 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:41.793683    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg" podStartSLOduration=1.7936688379999999 podStartE2EDuration="1.793668838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:56:41.792154426 +0000 UTC m=+2147.938604293" watchObservedRunningTime="2026-10-31 18:56:41.793668838 +0000 UTC m=+2147.940118700"
Oct 31 18:56:43.656329 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:43.656294    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:43.852704 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:43.852671    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:46.064980 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:46.064947    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:46.869140 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:46.869106    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:47.075015 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.074978    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:47.262885 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.262852    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"]
Oct 31 18:56:47.267057 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.267038    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"
Oct 31 18:56:47.324014 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.323981    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"]
Oct 31 18:56:47.370779 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.370741    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3932ef\""
Oct 31 18:56:47.371208 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.371184    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3932ef-dockercfg-xg8rp\""
Oct 31 18:56:47.406492 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.406447    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a7a922-9383-470e-8bcd-dcfc03082b29-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv\" (UID: \"23a7a922-9383-470e-8bcd-dcfc03082b29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"
Oct 31 18:56:47.406666 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.406509    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xr8b\" (UniqueName: \"kubernetes.io/projected/23a7a922-9383-470e-8bcd-dcfc03082b29-kube-api-access-4xr8b\") pod \"payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv\" (UID: \"23a7a922-9383-470e-8bcd-dcfc03082b29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"
Oct 31 18:56:47.424297 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.424266    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"]
Oct 31 18:56:47.428367 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.428350    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"
Oct 31 18:56:47.478893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.478864    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"]
Oct 31 18:56:47.507568 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.507534    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a7a922-9383-470e-8bcd-dcfc03082b29-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv\" (UID: \"23a7a922-9383-470e-8bcd-dcfc03082b29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"
Oct 31 18:56:47.507735 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.507624    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffqnf\" (UniqueName: \"kubernetes.io/projected/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-kube-api-access-ffqnf\") pod \"payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb\" (UID: \"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"
Oct 31 18:56:47.507735 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.507661    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xr8b\" (UniqueName: \"kubernetes.io/projected/23a7a922-9383-470e-8bcd-dcfc03082b29-kube-api-access-4xr8b\") pod \"payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv\" (UID: \"23a7a922-9383-470e-8bcd-dcfc03082b29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"
Oct 31 18:56:47.507873 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.507853    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb\" (UID: \"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"
Oct 31 18:56:47.508280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.508255    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a7a922-9383-470e-8bcd-dcfc03082b29-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv\" (UID: \"23a7a922-9383-470e-8bcd-dcfc03082b29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"
Oct 31 18:56:47.544219 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.544145    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"
Oct 31 18:56:47.544573 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.544556    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"
Oct 31 18:56:47.565880 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.565853    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xr8b\" (UniqueName: \"kubernetes.io/projected/23a7a922-9383-470e-8bcd-dcfc03082b29-kube-api-access-4xr8b\") pod \"payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv\" (UID: \"23a7a922-9383-470e-8bcd-dcfc03082b29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"
Oct 31 18:56:47.583694 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.583668    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"
Oct 31 18:56:47.609666 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.609632    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffqnf\" (UniqueName: \"kubernetes.io/projected/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-kube-api-access-ffqnf\") pod \"payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb\" (UID: \"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"
Oct 31 18:56:47.610752 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.609835    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb\" (UID: \"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"
Oct 31 18:56:47.610752 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.610704    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb\" (UID: \"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"
Oct 31 18:56:47.673089 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.673039    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffqnf\" (UniqueName: \"kubernetes.io/projected/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-kube-api-access-ffqnf\") pod \"payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb\" (UID: \"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"
Oct 31 18:56:47.742117 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.742087    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"
Oct 31 18:56:47.961202 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:47.961176    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"]
Oct 31 18:56:47.962069 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:56:47.962041    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a7a922_9383_470e_8bcd_dcfc03082b29.slice/crio-5dd0b938fb1163bd879e39d5be93700c7cd6a5261e2650a26b6c12f180bf3180 WatchSource:0}: Error finding container 5dd0b938fb1163bd879e39d5be93700c7cd6a5261e2650a26b6c12f180bf3180: Status 404 returned error can't find the container with id 5dd0b938fb1163bd879e39d5be93700c7cd6a5261e2650a26b6c12f180bf3180
Oct 31 18:56:48.116423 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:48.116398    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"]
Oct 31 18:56:48.118100 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:56:48.118076    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c5af01_fd8f_4f73_adbb_3ee549cd0ef6.slice/crio-2b7949e6aedb68ad9e7e8970f71a143a3268ebf76077108f09c607e9f854f953 WatchSource:0}: Error finding container 2b7949e6aedb68ad9e7e8970f71a143a3268ebf76077108f09c607e9f854f953: Status 404 returned error can't find the container with id 2b7949e6aedb68ad9e7e8970f71a143a3268ebf76077108f09c607e9f854f953
Oct 31 18:56:48.611242 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:48.611184    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb" event={"ID":"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6","Type":"ContainerStarted","Data":"398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693"}
Oct 31 18:56:48.611443 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:48.611249    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb" event={"ID":"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6","Type":"ContainerStarted","Data":"2b7949e6aedb68ad9e7e8970f71a143a3268ebf76077108f09c607e9f854f953"}
Oct 31 18:56:48.611443 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:48.611376    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"
Oct 31 18:56:48.612785 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:48.612762    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv" event={"ID":"23a7a922-9383-470e-8bcd-dcfc03082b29","Type":"ContainerStarted","Data":"0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27"}
Oct 31 18:56:48.612901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:48.612803    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"
Oct 31 18:56:48.612901 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:48.612818    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv" event={"ID":"23a7a922-9383-470e-8bcd-dcfc03082b29","Type":"ContainerStarted","Data":"5dd0b938fb1163bd879e39d5be93700c7cd6a5261e2650a26b6c12f180bf3180"}
Oct 31 18:56:48.729420 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:48.729372    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb" podStartSLOduration=1.729357467 podStartE2EDuration="1.729357467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:56:48.727577764 +0000 UTC m=+2154.874027627" watchObservedRunningTime="2026-10-31 18:56:48.729357467 +0000 UTC m=+2154.875807362"
Oct 31 18:56:48.843906 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:48.843835    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv" podStartSLOduration=1.84381575 podStartE2EDuration="1.84381575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:56:48.842497645 +0000 UTC m=+2154.988947502" watchObservedRunningTime="2026-10-31 18:56:48.84381575 +0000 UTC m=+2154.990265614"
Oct 31 18:56:52.151993 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:52.151962    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:52.364760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:52.364722    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:52.576806 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:52.576777    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"
Oct 31 18:56:52.576952 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:52.576839    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"
Oct 31 18:56:54.948079 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:54.948026    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:55.149791 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:55.149694    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:55.468888 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:56:55.468811    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:56:56.411820 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:56.411786    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:56.970760 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:56.970723    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:57.127324 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:57.127287    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:57.277708 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:57.277676    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:57.908237 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:57.908198    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:57.926750 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:57.926724    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:56:58.327934 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:58.327890    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:58.531343 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:58.531305    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:58.750430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:58.750397    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:59.179205 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:59.179126    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:59.328794 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:59.328766    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:56:59.622982 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:59.622957    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"
Oct 31 18:56:59.623161 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:56:59.623019    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"
Oct 31 18:57:02.212270 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:02.212247    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-62pcj_a44a1311-1de6-46c3-8070-07fa09d89854/payload-processing/0.log"
Oct 31 18:57:02.372144 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:02.372108    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:02.475924 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:02.475849    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-62pcj_a44a1311-1de6-46c3-8070-07fa09d89854/payload-processing/0.log"
Oct 31 18:57:04.195309 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:04.195276    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:04.416658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:04.416626    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:04.588610 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:04.588578    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:07.416197 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:07.416157    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:07.616868 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:07.616833    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:07.950559 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:07.950529    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"]
Oct 31 18:57:07.954505 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:07.954466    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"
Oct 31 18:57:08.009892 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.009865    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"]
Oct 31 18:57:08.037037 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.037003    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e34640-91c9-4f02-a50b-5a3da5a81f02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4\" (UID: \"12e34640-91c9-4f02-a50b-5a3da5a81f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"
Oct 31 18:57:08.037426 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.037397    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ll4\" (UniqueName: \"kubernetes.io/projected/12e34640-91c9-4f02-a50b-5a3da5a81f02-kube-api-access-66ll4\") pod \"payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4\" (UID: \"12e34640-91c9-4f02-a50b-5a3da5a81f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"
Oct 31 18:57:08.056790 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.056768    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-713bf0\""
Oct 31 18:57:08.056790 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.056783    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-713bf0-dockercfg-nj2k5\""
Oct 31 18:57:08.139130 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.139097    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ll4\" (UniqueName: \"kubernetes.io/projected/12e34640-91c9-4f02-a50b-5a3da5a81f02-kube-api-access-66ll4\") pod \"payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4\" (UID: \"12e34640-91c9-4f02-a50b-5a3da5a81f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"
Oct 31 18:57:08.139300 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.139171    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e34640-91c9-4f02-a50b-5a3da5a81f02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4\" (UID: \"12e34640-91c9-4f02-a50b-5a3da5a81f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"
Oct 31 18:57:08.140054 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.140031    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e34640-91c9-4f02-a50b-5a3da5a81f02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4\" (UID: \"12e34640-91c9-4f02-a50b-5a3da5a81f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"
Oct 31 18:57:08.195921 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.195889    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ll4\" (UniqueName: \"kubernetes.io/projected/12e34640-91c9-4f02-a50b-5a3da5a81f02-kube-api-access-66ll4\") pod \"payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4\" (UID: \"12e34640-91c9-4f02-a50b-5a3da5a81f02\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"
Oct 31 18:57:08.268118 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.268089    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"
Oct 31 18:57:08.640917 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.640892    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"]
Oct 31 18:57:08.642176 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:57:08.642142    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e34640_91c9_4f02_a50b_5a3da5a81f02.slice/crio-43b9ac6a5a4eaaefb0e732cf9dccc709698c8be28e81cb80b806ad7c30af0b38 WatchSource:0}: Error finding container 43b9ac6a5a4eaaefb0e732cf9dccc709698c8be28e81cb80b806ad7c30af0b38: Status 404 returned error can't find the container with id 43b9ac6a5a4eaaefb0e732cf9dccc709698c8be28e81cb80b806ad7c30af0b38
Oct 31 18:57:08.736062 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.736033    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4" event={"ID":"12e34640-91c9-4f02-a50b-5a3da5a81f02","Type":"ContainerStarted","Data":"28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5"}
Oct 31 18:57:08.736210 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.736080    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4" event={"ID":"12e34640-91c9-4f02-a50b-5a3da5a81f02","Type":"ContainerStarted","Data":"43b9ac6a5a4eaaefb0e732cf9dccc709698c8be28e81cb80b806ad7c30af0b38"}
Oct 31 18:57:08.736280 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.736230    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"
Oct 31 18:57:08.852136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:08.852083    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4" podStartSLOduration=1.8520697780000002 podStartE2EDuration="1.852069778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:57:08.850407192 +0000 UTC m=+2174.996857056" watchObservedRunningTime="2026-10-31 18:57:08.852069778 +0000 UTC m=+2174.998519676"
Oct 31 18:57:09.012226 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:09.012186    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:12.596344 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:12.596311    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:17.742006 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:17.741970    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:19.745626 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:19.745596    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"
Oct 31 18:57:22.737002 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:22.736967    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:22.886339 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:22.886300    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:23.044511 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:23.044406    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:25.468602 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:57:25.468568    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:57:26.548828 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:26.548791    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:57:27.839449 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:27.839414    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:28.075067 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:28.075033    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:32.855562 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:32.855523    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:39.157031 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:39.156951    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:43.744915 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:43.744883    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:43.923650 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:43.923615    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:44.078203 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:44.078109    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:48.584096 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:48.584058    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:52.694430 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:52.694389    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:53.405542 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:53.405504    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:55.467968 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:57:55.467937    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:57:56.868202 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:56.868157    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:57.191660 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:57.191562    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:57.352910 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:57.352879    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:57:58.436875 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:58.436837    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"]
Oct 31 18:57:58.440920 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:58.440902    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"
Oct 31 18:57:58.497318 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:58.497278    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"]
Oct 31 18:57:58.543607 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:58.543571    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-919b6e\""
Oct 31 18:57:58.543809 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:58.543638    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-919b6e-dockercfg-9tvft\""
Oct 31 18:57:58.584284 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:58.584251    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpsst\" (UniqueName: \"kubernetes.io/projected/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-kube-api-access-jpsst\") pod \"payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs\" (UID: \"fc20a9c6-8e4d-45ee-b309-39f286a35fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"
Oct 31 18:57:58.584467 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:58.584335    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs\" (UID: \"fc20a9c6-8e4d-45ee-b309-39f286a35fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"
Oct 31 18:57:58.685840 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:58.685803    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs\" (UID: \"fc20a9c6-8e4d-45ee-b309-39f286a35fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"
Oct 31 18:57:58.686043 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:58.685915    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsst\" (UniqueName: \"kubernetes.io/projected/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-kube-api-access-jpsst\") pod \"payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs\" (UID: \"fc20a9c6-8e4d-45ee-b309-39f286a35fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"
Oct 31 18:57:58.686595 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:58.686570    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs\" (UID: \"fc20a9c6-8e4d-45ee-b309-39f286a35fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"
Oct 31 18:57:58.743169 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:58.743147    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpsst\" (UniqueName: \"kubernetes.io/projected/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-kube-api-access-jpsst\") pod \"payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs\" (UID: \"fc20a9c6-8e4d-45ee-b309-39f286a35fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"
Oct 31 18:57:58.754966 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:58.754945    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"
Oct 31 18:57:59.126603 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:59.126574    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"]
Oct 31 18:57:59.132031 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:57:59.132004    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc20a9c6_8e4d_45ee_b309_39f286a35fe6.slice/crio-335158a25cb3020275fa6c3ec301a50445a711672a527513114e64931bf47e8e WatchSource:0}: Error finding container 335158a25cb3020275fa6c3ec301a50445a711672a527513114e64931bf47e8e: Status 404 returned error can't find the container with id 335158a25cb3020275fa6c3ec301a50445a711672a527513114e64931bf47e8e
Oct 31 18:57:59.873444 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:57:59.873408    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:00.044491 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:00.044442    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs" event={"ID":"fc20a9c6-8e4d-45ee-b309-39f286a35fe6","Type":"ContainerStarted","Data":"b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3"}
Oct 31 18:58:00.044658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:00.044615    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"
Oct 31 18:58:00.044700 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:00.044664    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs" event={"ID":"fc20a9c6-8e4d-45ee-b309-39f286a35fe6","Type":"ContainerStarted","Data":"335158a25cb3020275fa6c3ec301a50445a711672a527513114e64931bf47e8e"}
Oct 31 18:58:00.158639 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:00.158548    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs" podStartSLOduration=2.158531859 podStartE2EDuration="2.158531859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:58:00.158232151 +0000 UTC m=+2226.304682019" watchObservedRunningTime="2026-10-31 18:58:00.158531859 +0000 UTC m=+2226.304981723"
Oct 31 18:58:04.385223 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:04.385194    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:04.583252 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:04.583219    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:09.274900 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:09.274857    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:11.053524 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:11.053462    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"
Oct 31 18:58:13.964931 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:13.964892    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:17.042125 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:17.042094    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:58:18.596446 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:18.596411    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:23.220031 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:23.220000    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:25.469733 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:58:25.469700    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:58:27.830810 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:27.830757    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:32.544782 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:32.544734    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:34.318319 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:34.318286    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:37.806993 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:37.806958    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:38.233596 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:38.233513    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:41.369453 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:41.369415    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:42.555997 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:42.555963    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:43.173399 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:43.173302    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:43.318386 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:43.318355    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"]
Oct 31 18:58:43.398432 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:43.398391    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:43.424040 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:43.423968    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"]
Oct 31 18:58:43.688752 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:43.688663    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:43.846840 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:43.846807    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:44.446687 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:44.446651    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:44.595463 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:44.595421    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:47.568729 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:47.568696    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:47.744721 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:47.744663    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:47.921867 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:47.921776    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:48.319060 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.319024    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc" podUID="a63f4e31-36d7-4973-a471-8eba39524690" containerName="payload-processing" containerID="cri-o://52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1" gracePeriod=25
Oct 31 18:58:48.424580 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.424532    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67" podUID="9289a3ed-73c2-4ceb-9e79-56af282bb294" containerName="payload-pre-processing" containerID="cri-o://a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd" gracePeriod=25
Oct 31 18:58:48.587534 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.587441    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"
Oct 31 18:58:48.639160 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.639128    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63f4e31-36d7-4973-a471-8eba39524690-kube-api-access-vr2km\" (UniqueName: \"kubernetes.io/projected/a63f4e31-36d7-4973-a471-8eba39524690-kube-api-access-vr2km\") pod \"a63f4e31-36d7-4973-a471-8eba39524690\" (UID: \"a63f4e31-36d7-4973-a471-8eba39524690\") "
Oct 31 18:58:48.639323 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.639165    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63f4e31-36d7-4973-a471-8eba39524690-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a63f4e31-36d7-4973-a471-8eba39524690-plugins-config-volume\") pod \"a63f4e31-36d7-4973-a471-8eba39524690\" (UID: \"a63f4e31-36d7-4973-a471-8eba39524690\") "
Oct 31 18:58:48.639696 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.639641    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63f4e31-36d7-4973-a471-8eba39524690-plugins-config-volume" pod "a63f4e31-36d7-4973-a471-8eba39524690" (UID: "a63f4e31-36d7-4973-a471-8eba39524690"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:58:48.641324 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.641300    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63f4e31-36d7-4973-a471-8eba39524690-kube-api-access-vr2km" pod "a63f4e31-36d7-4973-a471-8eba39524690" (UID: "a63f4e31-36d7-4973-a471-8eba39524690"). InnerVolumeSpecName "kube-api-access-vr2km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:58:48.699452 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.699429    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"
Oct 31 18:58:48.739893 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.739867    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9289a3ed-73c2-4ceb-9e79-56af282bb294-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9289a3ed-73c2-4ceb-9e79-56af282bb294-plugins-config-volume\") pod \"9289a3ed-73c2-4ceb-9e79-56af282bb294\" (UID: \"9289a3ed-73c2-4ceb-9e79-56af282bb294\") "
Oct 31 18:58:48.740059 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.740043    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9289a3ed-73c2-4ceb-9e79-56af282bb294-kube-api-access-dnrwm\" (UniqueName: \"kubernetes.io/projected/9289a3ed-73c2-4ceb-9e79-56af282bb294-kube-api-access-dnrwm\") pod \"9289a3ed-73c2-4ceb-9e79-56af282bb294\" (UID: \"9289a3ed-73c2-4ceb-9e79-56af282bb294\") "
Oct 31 18:58:48.740296 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.740273    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9289a3ed-73c2-4ceb-9e79-56af282bb294-plugins-config-volume" pod "9289a3ed-73c2-4ceb-9e79-56af282bb294" (UID: "9289a3ed-73c2-4ceb-9e79-56af282bb294"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:58:48.740457 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.740440    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr2km\" (UniqueName: \"kubernetes.io/projected/a63f4e31-36d7-4973-a471-8eba39524690-kube-api-access-vr2km\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:58:48.740538 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.740464    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a63f4e31-36d7-4973-a471-8eba39524690-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:58:48.740538 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.740504    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9289a3ed-73c2-4ceb-9e79-56af282bb294-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:58:48.742265 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.742243    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9289a3ed-73c2-4ceb-9e79-56af282bb294-kube-api-access-dnrwm" pod "9289a3ed-73c2-4ceb-9e79-56af282bb294" (UID: "9289a3ed-73c2-4ceb-9e79-56af282bb294"). InnerVolumeSpecName "kube-api-access-dnrwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:58:48.841981 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:48.841914    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnrwm\" (UniqueName: \"kubernetes.io/projected/9289a3ed-73c2-4ceb-9e79-56af282bb294-kube-api-access-dnrwm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:58:49.337495 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.337439    4484 generic.go:356] "Generic (PLEG): container finished" podID="9289a3ed-73c2-4ceb-9e79-56af282bb294" containerID="a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd" exitCode=0
Oct 31 18:58:49.337635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.337524    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"
Oct 31 18:58:49.337635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.337528    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67" event={"ID":"9289a3ed-73c2-4ceb-9e79-56af282bb294","Type":"ContainerDied","Data":"a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd"}
Oct 31 18:58:49.337635 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.337606    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67" event={"ID":"9289a3ed-73c2-4ceb-9e79-56af282bb294","Type":"ContainerDied","Data":"e62bec0910e1331039c907b1381c8d6d3ab8284640279c59301c4f524ccbaba3"}
Oct 31 18:58:49.337793 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.337638    4484 scope.go:122] "RemoveContainer" containerID="a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd"
Oct 31 18:58:49.339089 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.338988    4484 generic.go:356] "Generic (PLEG): container finished" podID="a63f4e31-36d7-4973-a471-8eba39524690" containerID="52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1" exitCode=0
Oct 31 18:58:49.339089 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.339020    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc" event={"ID":"a63f4e31-36d7-4973-a471-8eba39524690","Type":"ContainerDied","Data":"52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1"}
Oct 31 18:58:49.339089 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.339084    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc" event={"ID":"a63f4e31-36d7-4973-a471-8eba39524690","Type":"ContainerDied","Data":"8813b728701940a55fbc1f59573d470315548bfaf6a820a05f7c31cb2ae11dd2"}
Oct 31 18:58:49.339293 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.339031    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"
Oct 31 18:58:49.348921 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.348898    4484 scope.go:122] "RemoveContainer" containerID="a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd"
Oct 31 18:58:49.349350 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:58:49.349248    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd\": container with ID starting with a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd not found: ID does not exist" containerID="a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd"
Oct 31 18:58:49.349350 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.349275    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd"} err="failed to get container status \"a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd\": rpc error: code = NotFound desc = could not find container \"a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd\": container with ID starting with a992d3b852f421e903f83e3f0cf4880f3f71c08218fc722a8aff9ade6c2c98fd not found: ID does not exist"
Oct 31 18:58:49.349350 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.349298    4484 scope.go:122] "RemoveContainer" containerID="52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1"
Oct 31 18:58:49.358159 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.358142    4484 scope.go:122] "RemoveContainer" containerID="52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1"
Oct 31 18:58:49.358419 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:58:49.358397    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1\": container with ID starting with 52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1 not found: ID does not exist" containerID="52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1"
Oct 31 18:58:49.358544 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.358429    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1"} err="failed to get container status \"52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1\": rpc error: code = NotFound desc = could not find container \"52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1\": container with ID starting with 52fe229ca13824afbdac560bf32fabe0c67343065b53413e2f8334063463b4a1 not found: ID does not exist"
Oct 31 18:58:49.502843 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.502810    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"]
Oct 31 18:58:49.553400 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.553369    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-84c46e-74648574d7-6rx67"]
Oct 31 18:58:49.626613 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.626517    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"]
Oct 31 18:58:49.632784 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.632754    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-84c46e-79cdbc78cf-m6jgc"]
Oct 31 18:58:49.717507 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.717457    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:49.994497 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.994451    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9289a3ed-73c2-4ceb-9e79-56af282bb294" path="/var/lib/kubelet/pods/9289a3ed-73c2-4ceb-9e79-56af282bb294/volumes"
Oct 31 18:58:49.994983 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:49.994962    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63f4e31-36d7-4973-a471-8eba39524690" path="/var/lib/kubelet/pods/a63f4e31-36d7-4973-a471-8eba39524690/volumes"
Oct 31 18:58:50.461837 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:50.461750    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:50.750118 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:50.750076    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:50.935744 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:50.935714    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:51.151704 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.151611    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"]
Oct 31 18:58:51.152287 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.152267    4484 cpu_manager.go:414] "containerMap: removing container" podUID="9289a3ed-73c2-4ceb-9e79-56af282bb294" containerName="payload-pre-processing"
Oct 31 18:58:51.152387 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.152289    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9289a3ed-73c2-4ceb-9e79-56af282bb294" containerName="payload-pre-processing"
Oct 31 18:58:51.152387 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.152314    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a63f4e31-36d7-4973-a471-8eba39524690" containerName="payload-processing"
Oct 31 18:58:51.152387 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.152321    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63f4e31-36d7-4973-a471-8eba39524690" containerName="payload-processing"
Oct 31 18:58:51.152589 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.152504    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9289a3ed-73c2-4ceb-9e79-56af282bb294" containerName="payload-pre-processing"
Oct 31 18:58:51.152589 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.152520    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63f4e31-36d7-4973-a471-8eba39524690" containerName="payload-processing"
Oct 31 18:58:51.157209 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.157189    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"
Oct 31 18:58:51.211004 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.210973    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"]
Oct 31 18:58:51.258132 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.258104    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e43ccf\""
Oct 31 18:58:51.269091 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.269042    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091ccab3-9990-4362-a441-a4d8a8782b6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8\" (UID: \"091ccab3-9990-4362-a441-a4d8a8782b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"
Oct 31 18:58:51.275539 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.275506    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gs2n\" (UniqueName: \"kubernetes.io/projected/091ccab3-9990-4362-a441-a4d8a8782b6b-kube-api-access-2gs2n\") pod \"payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8\" (UID: \"091ccab3-9990-4362-a441-a4d8a8782b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"
Oct 31 18:58:51.307147 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.307114    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"]
Oct 31 18:58:51.311258 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.311235    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"
Oct 31 18:58:51.364265 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.364237    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e43ccf-dockercfg-wxvsj\""
Oct 31 18:58:51.366090 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.366061    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"]
Oct 31 18:58:51.376458 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.376430    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp\" (UID: \"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"
Oct 31 18:58:51.376620 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.376466    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cflzf\" (UniqueName: \"kubernetes.io/projected/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-kube-api-access-cflzf\") pod \"payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp\" (UID: \"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"
Oct 31 18:58:51.376620 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.376605    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gs2n\" (UniqueName: \"kubernetes.io/projected/091ccab3-9990-4362-a441-a4d8a8782b6b-kube-api-access-2gs2n\") pod \"payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8\" (UID: \"091ccab3-9990-4362-a441-a4d8a8782b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"
Oct 31 18:58:51.376794 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.376771    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091ccab3-9990-4362-a441-a4d8a8782b6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8\" (UID: \"091ccab3-9990-4362-a441-a4d8a8782b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"
Oct 31 18:58:51.377696 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.377671    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091ccab3-9990-4362-a441-a4d8a8782b6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8\" (UID: \"091ccab3-9990-4362-a441-a4d8a8782b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"
Oct 31 18:58:51.434070 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.434007    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gs2n\" (UniqueName: \"kubernetes.io/projected/091ccab3-9990-4362-a441-a4d8a8782b6b-kube-api-access-2gs2n\") pod \"payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8\" (UID: \"091ccab3-9990-4362-a441-a4d8a8782b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"
Oct 31 18:58:51.470099 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.470069    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"
Oct 31 18:58:51.477688 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.477656    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp\" (UID: \"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"
Oct 31 18:58:51.477802 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.477706    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cflzf\" (UniqueName: \"kubernetes.io/projected/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-kube-api-access-cflzf\") pod \"payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp\" (UID: \"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"
Oct 31 18:58:51.478351 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.478335    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp\" (UID: \"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"
Oct 31 18:58:51.538350 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.538027    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cflzf\" (UniqueName: \"kubernetes.io/projected/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-kube-api-access-cflzf\") pod \"payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp\" (UID: \"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"
Oct 31 18:58:51.624571 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.624542    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"
Oct 31 18:58:51.843647 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:51.843616    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"]
Oct 31 18:58:51.844787 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:58:51.844761    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091ccab3_9990_4362_a441_a4d8a8782b6b.slice/crio-75ef37bd265ad47a55dfadc2a05cdfda39df9acbc88c7f16e3d256360b6f2df7 WatchSource:0}: Error finding container 75ef37bd265ad47a55dfadc2a05cdfda39df9acbc88c7f16e3d256360b6f2df7: Status 404 returned error can't find the container with id 75ef37bd265ad47a55dfadc2a05cdfda39df9acbc88c7f16e3d256360b6f2df7
Oct 31 18:58:52.002610 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:52.002585    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"]
Oct 31 18:58:52.003817 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:58:52.003792    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169ef4d7_7ac6_4436_a7fe_f9f2a951d5fc.slice/crio-cabcddbc24e0f09475ea8bea12425eb5bf537ccf79e730f4b77a9d9428978098 WatchSource:0}: Error finding container cabcddbc24e0f09475ea8bea12425eb5bf537ccf79e730f4b77a9d9428978098: Status 404 returned error can't find the container with id cabcddbc24e0f09475ea8bea12425eb5bf537ccf79e730f4b77a9d9428978098
Oct 31 18:58:52.358879 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:52.358778    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8" event={"ID":"091ccab3-9990-4362-a441-a4d8a8782b6b","Type":"ContainerStarted","Data":"1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2"}
Oct 31 18:58:52.358879 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:52.358842    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8" event={"ID":"091ccab3-9990-4362-a441-a4d8a8782b6b","Type":"ContainerStarted","Data":"75ef37bd265ad47a55dfadc2a05cdfda39df9acbc88c7f16e3d256360b6f2df7"}
Oct 31 18:58:52.359114 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:52.358974    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"
Oct 31 18:58:52.360406 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:52.360380    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp" event={"ID":"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc","Type":"ContainerStarted","Data":"11c043720de85c97383c8a0c355f5e4a5b54f5b01108b80e8570b224e6b21269"}
Oct 31 18:58:52.360561 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:52.360421    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp" event={"ID":"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc","Type":"ContainerStarted","Data":"cabcddbc24e0f09475ea8bea12425eb5bf537ccf79e730f4b77a9d9428978098"}
Oct 31 18:58:52.360561 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:52.360447    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"
Oct 31 18:58:52.588245 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:52.588029    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8" podStartSLOduration=1.588014359 podStartE2EDuration="1.588014359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:58:52.472606463 +0000 UTC m=+2278.619056327" watchObservedRunningTime="2026-10-31 18:58:52.588014359 +0000 UTC m=+2278.734464232"
Oct 31 18:58:52.591050 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:52.589969    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp" podStartSLOduration=1.5899524569999999 podStartE2EDuration="1.589952457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:58:52.586787548 +0000 UTC m=+2278.733237413" watchObservedRunningTime="2026-10-31 18:58:52.589952457 +0000 UTC m=+2278.736402320"
Oct 31 18:58:52.665864 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:52.665786    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:52.855286 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:52.855248    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:53.025891 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:53.025861    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:54.777184 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:54.777148    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:55.467953 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:58:55.467919    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:58:56.551913 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:56.551882    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:57.528615 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:57.528569    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:57.704246 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:57.704212    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:58.433527 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:58.433455    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:58:58.679890 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:58:58.679854    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:00.062634 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:00.062600    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:02.895658 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:02.895620    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:03.052051 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:03.052008    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:03.204061 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:03.203967    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:03.369563 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:03.369530    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"
Oct 31 18:59:03.369956 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:03.369941    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"
Oct 31 18:59:06.133042 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.132859    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:06.305676 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.305645    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:06.518087 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.518056    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"]
Oct 31 18:59:06.523316 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.523294    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"
Oct 31 18:59:06.576974 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.576945    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"]
Oct 31 18:59:06.624128 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.624094    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b11e44\""
Oct 31 18:59:06.624294 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.624129    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b11e44-dockercfg-z78cc\""
Oct 31 18:59:06.652355 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.652323    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn46k\" (UniqueName: \"kubernetes.io/projected/8ed1f34a-89a9-4dca-a574-e602d5471ed3-kube-api-access-rn46k\") pod \"payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht\" (UID: \"8ed1f34a-89a9-4dca-a574-e602d5471ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"
Oct 31 18:59:06.652542 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.652367    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed1f34a-89a9-4dca-a574-e602d5471ed3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht\" (UID: \"8ed1f34a-89a9-4dca-a574-e602d5471ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"
Oct 31 18:59:06.677532 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.677508    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"]
Oct 31 18:59:06.681424 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.681410    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"
Oct 31 18:59:06.727503 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.727445    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"]
Oct 31 18:59:06.753271 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.753245    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn46k\" (UniqueName: \"kubernetes.io/projected/8ed1f34a-89a9-4dca-a574-e602d5471ed3-kube-api-access-rn46k\") pod \"payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht\" (UID: \"8ed1f34a-89a9-4dca-a574-e602d5471ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"
Oct 31 18:59:06.753433 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.753305    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed1f34a-89a9-4dca-a574-e602d5471ed3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht\" (UID: \"8ed1f34a-89a9-4dca-a574-e602d5471ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"
Oct 31 18:59:06.753943 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.753924    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed1f34a-89a9-4dca-a574-e602d5471ed3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht\" (UID: \"8ed1f34a-89a9-4dca-a574-e602d5471ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"
Oct 31 18:59:06.810810 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.810742    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn46k\" (UniqueName: \"kubernetes.io/projected/8ed1f34a-89a9-4dca-a574-e602d5471ed3-kube-api-access-rn46k\") pod \"payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht\" (UID: \"8ed1f34a-89a9-4dca-a574-e602d5471ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"
Oct 31 18:59:06.829817 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.829783    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:06.836633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.836611    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"
Oct 31 18:59:06.855767 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.855715    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvp74\" (UniqueName: \"kubernetes.io/projected/34bac501-dd5d-499e-80e4-e7016b536662-kube-api-access-lvp74\") pod \"payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4\" (UID: \"34bac501-dd5d-499e-80e4-e7016b536662\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"
Oct 31 18:59:06.856127 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.855782    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34bac501-dd5d-499e-80e4-e7016b536662-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4\" (UID: \"34bac501-dd5d-499e-80e4-e7016b536662\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"
Oct 31 18:59:06.956740 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.956709    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvp74\" (UniqueName: \"kubernetes.io/projected/34bac501-dd5d-499e-80e4-e7016b536662-kube-api-access-lvp74\") pod \"payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4\" (UID: \"34bac501-dd5d-499e-80e4-e7016b536662\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"
Oct 31 18:59:06.956911 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.956753    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34bac501-dd5d-499e-80e4-e7016b536662-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4\" (UID: \"34bac501-dd5d-499e-80e4-e7016b536662\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"
Oct 31 18:59:06.957531 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:06.957505    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34bac501-dd5d-499e-80e4-e7016b536662-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4\" (UID: \"34bac501-dd5d-499e-80e4-e7016b536662\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"
Oct 31 18:59:07.013920 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:07.013892    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvp74\" (UniqueName: \"kubernetes.io/projected/34bac501-dd5d-499e-80e4-e7016b536662-kube-api-access-lvp74\") pod \"payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4\" (UID: \"34bac501-dd5d-499e-80e4-e7016b536662\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"
Oct 31 18:59:07.215800 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:07.215774    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"]
Oct 31 18:59:07.216207 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:59:07.215970    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed1f34a_89a9_4dca_a574_e602d5471ed3.slice/crio-ea67d41eb108193ad5cfc8534b7852c6be21757f5f52f37f6ada695e17e68e84 WatchSource:0}: Error finding container ea67d41eb108193ad5cfc8534b7852c6be21757f5f52f37f6ada695e17e68e84: Status 404 returned error can't find the container with id ea67d41eb108193ad5cfc8534b7852c6be21757f5f52f37f6ada695e17e68e84
Oct 31 18:59:07.294114 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:07.294089    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"
Oct 31 18:59:07.459597 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:07.459561    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht" event={"ID":"8ed1f34a-89a9-4dca-a574-e602d5471ed3","Type":"ContainerStarted","Data":"512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455"}
Oct 31 18:59:07.459769 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:07.459632    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht" event={"ID":"8ed1f34a-89a9-4dca-a574-e602d5471ed3","Type":"ContainerStarted","Data":"ea67d41eb108193ad5cfc8534b7852c6be21757f5f52f37f6ada695e17e68e84"}
Oct 31 18:59:07.459769 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:07.459745    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"
Oct 31 18:59:07.576488 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:07.576423    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht" podStartSLOduration=1.576410875 podStartE2EDuration="1.576410875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:59:07.573568714 +0000 UTC m=+2293.720018578" watchObservedRunningTime="2026-10-31 18:59:07.576410875 +0000 UTC m=+2293.722860739"
Oct 31 18:59:07.662995 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:07.662967    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"]
Oct 31 18:59:07.663632 ip-10-0-142-120 kubenswrapper[4484]: W1031 18:59:07.663606    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bac501_dd5d_499e_80e4_e7016b536662.slice/crio-5c8dd38cb55206a78b805c03fbed7188c27688837b5d45dfee6fa73cdf12bcbc WatchSource:0}: Error finding container 5c8dd38cb55206a78b805c03fbed7188c27688837b5d45dfee6fa73cdf12bcbc: Status 404 returned error can't find the container with id 5c8dd38cb55206a78b805c03fbed7188c27688837b5d45dfee6fa73cdf12bcbc
Oct 31 18:59:08.467535 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:08.467497    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4" event={"ID":"34bac501-dd5d-499e-80e4-e7016b536662","Type":"ContainerStarted","Data":"c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54"}
Oct 31 18:59:08.467919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:08.467546    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4" event={"ID":"34bac501-dd5d-499e-80e4-e7016b536662","Type":"ContainerStarted","Data":"5c8dd38cb55206a78b805c03fbed7188c27688837b5d45dfee6fa73cdf12bcbc"}
Oct 31 18:59:08.467919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:08.467574    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"
Oct 31 18:59:08.582271 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:08.582216    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4" podStartSLOduration=2.582199423 podStartE2EDuration="2.582199423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 18:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 18:59:08.581791323 +0000 UTC m=+2294.728241186" watchObservedRunningTime="2026-10-31 18:59:08.582199423 +0000 UTC m=+2294.728649287"
Oct 31 18:59:11.858995 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:11.858954    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:16.663825 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:16.663790    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:18.470633 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:18.470609    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"
Oct 31 18:59:19.476423 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:19.476391    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"
Oct 31 18:59:20.845880 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:20.845842    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"]
Oct 31 18:59:20.968695 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:20.968665    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:21.175776 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:21.175695    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:21.339491 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:21.339441    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:21.498258 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:21.498226    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:22.721102 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:22.721065    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:22.922601 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:22.922565    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:24.243678 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:24.243643    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:24.740594 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:24.740548    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:25.468331 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:59:25.468295    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:59:25.535347 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:25.535314    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:25.846733 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:25.846671    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4" podUID="12e34640-91c9-4f02-a50b-5a3da5a81f02" containerName="payload-processing" containerID="cri-o://28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5" gracePeriod=25
Oct 31 18:59:26.119846 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.119816    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"
Oct 31 18:59:26.163919 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.163889    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e34640-91c9-4f02-a50b-5a3da5a81f02-kube-api-access-66ll4\" (UniqueName: \"kubernetes.io/projected/12e34640-91c9-4f02-a50b-5a3da5a81f02-kube-api-access-66ll4\") pod \"12e34640-91c9-4f02-a50b-5a3da5a81f02\" (UID: \"12e34640-91c9-4f02-a50b-5a3da5a81f02\") "
Oct 31 18:59:26.164091 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.163940    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e34640-91c9-4f02-a50b-5a3da5a81f02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e34640-91c9-4f02-a50b-5a3da5a81f02-plugins-config-volume\") pod \"12e34640-91c9-4f02-a50b-5a3da5a81f02\" (UID: \"12e34640-91c9-4f02-a50b-5a3da5a81f02\") "
Oct 31 18:59:26.164322 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.164302    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e34640-91c9-4f02-a50b-5a3da5a81f02-plugins-config-volume" pod "12e34640-91c9-4f02-a50b-5a3da5a81f02" (UID: "12e34640-91c9-4f02-a50b-5a3da5a81f02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:59:26.166225 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.166201    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e34640-91c9-4f02-a50b-5a3da5a81f02-kube-api-access-66ll4" pod "12e34640-91c9-4f02-a50b-5a3da5a81f02" (UID: "12e34640-91c9-4f02-a50b-5a3da5a81f02"). InnerVolumeSpecName "kube-api-access-66ll4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:59:26.265721 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.265685    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66ll4\" (UniqueName: \"kubernetes.io/projected/12e34640-91c9-4f02-a50b-5a3da5a81f02-kube-api-access-66ll4\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:26.265721 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.265711    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e34640-91c9-4f02-a50b-5a3da5a81f02-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:26.590483 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.590439    4484 generic.go:356] "Generic (PLEG): container finished" podID="12e34640-91c9-4f02-a50b-5a3da5a81f02" containerID="28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5" exitCode=0
Oct 31 18:59:26.590897 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.590509    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"
Oct 31 18:59:26.590897 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.590539    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4" event={"ID":"12e34640-91c9-4f02-a50b-5a3da5a81f02","Type":"ContainerDied","Data":"28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5"}
Oct 31 18:59:26.590897 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.590601    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4" event={"ID":"12e34640-91c9-4f02-a50b-5a3da5a81f02","Type":"ContainerDied","Data":"43b9ac6a5a4eaaefb0e732cf9dccc709698c8be28e81cb80b806ad7c30af0b38"}
Oct 31 18:59:26.590897 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.590630    4484 scope.go:122] "RemoveContainer" containerID="28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5"
Oct 31 18:59:26.601749 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.601728    4484 scope.go:122] "RemoveContainer" containerID="28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5"
Oct 31 18:59:26.602050 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:59:26.602031    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5\": container with ID starting with 28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5 not found: ID does not exist" containerID="28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5"
Oct 31 18:59:26.602109 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.602058    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5"} err="failed to get container status \"28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5\": rpc error: code = NotFound desc = could not find container \"28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5\": container with ID starting with 28336adeeac69686fea66ac547cdb08a29d380aa7a6a8d277616115d8a7253a5 not found: ID does not exist"
Oct 31 18:59:26.758739 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.758694    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"]
Oct 31 18:59:26.808958 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:26.808930    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-713bf0-5546f5689f-8v4z4"]
Oct 31 18:59:27.743846 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:27.743813    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:27.993743 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:27.993714    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e34640-91c9-4f02-a50b-5a3da5a81f02" path="/var/lib/kubelet/pods/12e34640-91c9-4f02-a50b-5a3da5a81f02/volumes"
Oct 31 18:59:30.811642 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:30.811609    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"]
Oct 31 18:59:30.918187 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:30.918156    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"]
Oct 31 18:59:31.187298 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:31.187219    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:31.358758 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:31.358720    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:31.532557 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:31.532518    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:32.012717 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:32.012680    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:33.864844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:33.864798    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:35.812915 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:35.812859    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q" podUID="d4904ff6-8feb-4491-9503-52734d32c17c" containerName="payload-processing" containerID="cri-o://86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d" gracePeriod=25
Oct 31 18:59:35.919150 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:35.919114    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh" podUID="659c1e71-ce7c-4d01-8340-2263f57438ea" containerName="payload-pre-processing" containerID="cri-o://bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0" gracePeriod=25
Oct 31 18:59:36.076318 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.076259    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"
Oct 31 18:59:36.178882 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.178850    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4904ff6-8feb-4491-9503-52734d32c17c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4904ff6-8feb-4491-9503-52734d32c17c-plugins-config-volume\") pod \"d4904ff6-8feb-4491-9503-52734d32c17c\" (UID: \"d4904ff6-8feb-4491-9503-52734d32c17c\") "
Oct 31 18:59:36.179041 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.179008    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4904ff6-8feb-4491-9503-52734d32c17c-kube-api-access-qrggv\" (UniqueName: \"kubernetes.io/projected/d4904ff6-8feb-4491-9503-52734d32c17c-kube-api-access-qrggv\") pod \"d4904ff6-8feb-4491-9503-52734d32c17c\" (UID: \"d4904ff6-8feb-4491-9503-52734d32c17c\") "
Oct 31 18:59:36.179211 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.179187    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4904ff6-8feb-4491-9503-52734d32c17c-plugins-config-volume" pod "d4904ff6-8feb-4491-9503-52734d32c17c" (UID: "d4904ff6-8feb-4491-9503-52734d32c17c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:59:36.179410 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.179393    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4904ff6-8feb-4491-9503-52734d32c17c-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:36.181254 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.181235    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4904ff6-8feb-4491-9503-52734d32c17c-kube-api-access-qrggv" pod "d4904ff6-8feb-4491-9503-52734d32c17c" (UID: "d4904ff6-8feb-4491-9503-52734d32c17c"). InnerVolumeSpecName "kube-api-access-qrggv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:59:36.200572 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.200550    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"
Oct 31 18:59:36.279773 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.279743    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659c1e71-ce7c-4d01-8340-2263f57438ea-kube-api-access-khk8v\" (UniqueName: \"kubernetes.io/projected/659c1e71-ce7c-4d01-8340-2263f57438ea-kube-api-access-khk8v\") pod \"659c1e71-ce7c-4d01-8340-2263f57438ea\" (UID: \"659c1e71-ce7c-4d01-8340-2263f57438ea\") "
Oct 31 18:59:36.279959 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.279930    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659c1e71-ce7c-4d01-8340-2263f57438ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659c1e71-ce7c-4d01-8340-2263f57438ea-plugins-config-volume\") pod \"659c1e71-ce7c-4d01-8340-2263f57438ea\" (UID: \"659c1e71-ce7c-4d01-8340-2263f57438ea\") "
Oct 31 18:59:36.280985 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.280883    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659c1e71-ce7c-4d01-8340-2263f57438ea-plugins-config-volume" pod "659c1e71-ce7c-4d01-8340-2263f57438ea" (UID: "659c1e71-ce7c-4d01-8340-2263f57438ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 18:59:36.289138 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.281658    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrggv\" (UniqueName: \"kubernetes.io/projected/d4904ff6-8feb-4491-9503-52734d32c17c-kube-api-access-qrggv\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:36.289138 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.281689    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/659c1e71-ce7c-4d01-8340-2263f57438ea-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:36.289334 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.289255    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659c1e71-ce7c-4d01-8340-2263f57438ea-kube-api-access-khk8v" pod "659c1e71-ce7c-4d01-8340-2263f57438ea" (UID: "659c1e71-ce7c-4d01-8340-2263f57438ea"). InnerVolumeSpecName "kube-api-access-khk8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 18:59:36.360263 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.360192    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 18:59:36.383207 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.383175    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khk8v\" (UniqueName: \"kubernetes.io/projected/659c1e71-ce7c-4d01-8340-2263f57438ea-kube-api-access-khk8v\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 18:59:36.655866 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.655776    4484 generic.go:356] "Generic (PLEG): container finished" podID="d4904ff6-8feb-4491-9503-52734d32c17c" containerID="86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d" exitCode=0
Oct 31 18:59:36.655866 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.655846    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"
Oct 31 18:59:36.655866 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.655854    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q" event={"ID":"d4904ff6-8feb-4491-9503-52734d32c17c","Type":"ContainerDied","Data":"86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d"}
Oct 31 18:59:36.656142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.655902    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q" event={"ID":"d4904ff6-8feb-4491-9503-52734d32c17c","Type":"ContainerDied","Data":"5408f107ac9dcfd931918525094f83c5daacb4cad8983b9e19d3fd4071c86df9"}
Oct 31 18:59:36.656142 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.655916    4484 scope.go:122] "RemoveContainer" containerID="86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d"
Oct 31 18:59:36.657358 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.657332    4484 generic.go:356] "Generic (PLEG): container finished" podID="659c1e71-ce7c-4d01-8340-2263f57438ea" containerID="bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0" exitCode=0
Oct 31 18:59:36.657519 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.657364    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh" event={"ID":"659c1e71-ce7c-4d01-8340-2263f57438ea","Type":"ContainerDied","Data":"bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0"}
Oct 31 18:59:36.657519 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.657411    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"
Oct 31 18:59:36.657519 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.657413    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh" event={"ID":"659c1e71-ce7c-4d01-8340-2263f57438ea","Type":"ContainerDied","Data":"e18872d112bbc76362240842301946af9d59426bdae9cbee99b2436ce3db87ba"}
Oct 31 18:59:36.665648 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.665633    4484 scope.go:122] "RemoveContainer" containerID="86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d"
Oct 31 18:59:36.665951 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:59:36.665929    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d\": container with ID starting with 86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d not found: ID does not exist" containerID="86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d"
Oct 31 18:59:36.666058 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.665957    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d"} err="failed to get container status \"86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d\": rpc error: code = NotFound desc = could not find container \"86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d\": container with ID starting with 86a7d4fec048a7f7eabf77819371e7aa97d2a42715c3a8fab69255a9024d2a1d not found: ID does not exist"
Oct 31 18:59:36.666058 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.665979    4484 scope.go:122] "RemoveContainer" containerID="bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0"
Oct 31 18:59:36.674811 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.674795    4484 scope.go:122] "RemoveContainer" containerID="bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0"
Oct 31 18:59:36.675063 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:59:36.675046    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0\": container with ID starting with bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0 not found: ID does not exist" containerID="bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0"
Oct 31 18:59:36.675112 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.675070    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0"} err="failed to get container status \"bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0\": rpc error: code = NotFound desc = could not find container \"bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0\": container with ID starting with bc43a388b9ec1339f0e88d46d56d041caa6d88175824adb9d100d7df17c21be0 not found: ID does not exist"
Oct 31 18:59:36.821907 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.821867    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"]
Oct 31 18:59:36.826844 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.826818    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-35566a-86f46cb97c-n9s2q"]
Oct 31 18:59:36.936606 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.936523    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"]
Oct 31 18:59:36.939136 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:36.939116    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-35566a-76765b96f5-jftjh"]
Oct 31 18:59:37.227073 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:37.226992    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:37.994968 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:37.994938    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659c1e71-ce7c-4d01-8340-2263f57438ea" path="/var/lib/kubelet/pods/659c1e71-ce7c-4d01-8340-2263f57438ea/volumes"
Oct 31 18:59:37.995361 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:37.995298    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4904ff6-8feb-4491-9503-52734d32c17c" path="/var/lib/kubelet/pods/d4904ff6-8feb-4491-9503-52734d32c17c/volumes"
Oct 31 18:59:39.067253 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:39.067222    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:45.186515 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:45.186487    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp_16186434-2291-4ba4-bd98-ddcfa937eff6/payload-processing/0.log"
Oct 31 18:59:45.285836 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:45.285799    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:45.476152 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:45.476075    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-62pcj_a44a1311-1de6-46c3-8070-07fa09d89854/payload-processing/0.log"
Oct 31 18:59:45.719784 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:45.719738    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-62pcj_a44a1311-1de6-46c3-8070-07fa09d89854/payload-processing/0.log"
Oct 31 18:59:47.204203 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:47.204168    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:49.850162 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:49.850122    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:49.999346 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:49.999315    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:50.165857 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:50.165766    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:52.231192 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:52.231147    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:53.372461 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:53.372424    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:54.714250 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:54.714214    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:55.467923 ip-10-0-142-120 kubenswrapper[4484]: E1031 18:59:55.467889    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 18:59:57.857682 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:57.857648    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 18:59:59.471211 ip-10-0-142-120 kubenswrapper[4484]: I1031 18:59:59.471178    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:00.161307 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.161269    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"]
Oct 31 19:00:00.161927 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.161907    4484 cpu_manager.go:414] "containerMap: removing container" podUID="659c1e71-ce7c-4d01-8340-2263f57438ea" containerName="payload-pre-processing"
Oct 31 19:00:00.162040 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.161928    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659c1e71-ce7c-4d01-8340-2263f57438ea" containerName="payload-pre-processing"
Oct 31 19:00:00.162040 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.161954    4484 cpu_manager.go:414] "containerMap: removing container" podUID="d4904ff6-8feb-4491-9503-52734d32c17c" containerName="payload-processing"
Oct 31 19:00:00.162040 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.161963    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4904ff6-8feb-4491-9503-52734d32c17c" containerName="payload-processing"
Oct 31 19:00:00.162040 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.161974    4484 cpu_manager.go:414] "containerMap: removing container" podUID="12e34640-91c9-4f02-a50b-5a3da5a81f02" containerName="payload-processing"
Oct 31 19:00:00.162040 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.161980    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e34640-91c9-4f02-a50b-5a3da5a81f02" containerName="payload-processing"
Oct 31 19:00:00.162309 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.162154    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="659c1e71-ce7c-4d01-8340-2263f57438ea" containerName="payload-pre-processing"
Oct 31 19:00:00.162309 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.162173    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4904ff6-8feb-4491-9503-52734d32c17c" containerName="payload-processing"
Oct 31 19:00:00.162309 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.162181    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e34640-91c9-4f02-a50b-5a3da5a81f02" containerName="payload-processing"
Oct 31 19:00:00.166750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.166728    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"]
Oct 31 19:00:00.166875 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.166835    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"
Oct 31 19:00:00.170685 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.170667    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"
Oct 31 19:00:00.187267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.187248    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"]
Oct 31 19:00:00.190884 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.190870    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"
Oct 31 19:00:00.211374 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.211351    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"]
Oct 31 19:00:00.215849 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.215833    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"]
Oct 31 19:00:00.215945 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.215929    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"
Oct 31 19:00:00.220145 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.220129    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"]
Oct 31 19:00:00.220252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.220239    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"
Oct 31 19:00:00.223869 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.223853    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"
Oct 31 19:00:00.235272 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.235244    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7t27\" (UniqueName: \"kubernetes.io/projected/55018466-4443-4d03-a113-c05f8c96980f-kube-api-access-n7t27\") pod \"maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks\" (UID: \"55018466-4443-4d03-a113-c05f8c96980f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"
Oct 31 19:00:00.235380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.235290    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f10ee1e-021c-45ea-b301-a50639c9731d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq\" (UID: \"5f10ee1e-021c-45ea-b301-a50639c9731d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"
Oct 31 19:00:00.235448 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.235404    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55018466-4443-4d03-a113-c05f8c96980f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks\" (UID: \"55018466-4443-4d03-a113-c05f8c96980f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"
Oct 31 19:00:00.235537 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.235452    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh5rh\" (UniqueName: \"kubernetes.io/projected/5f10ee1e-021c-45ea-b301-a50639c9731d-kube-api-access-gh5rh\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq\" (UID: \"5f10ee1e-021c-45ea-b301-a50639c9731d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"
Oct 31 19:00:00.235583 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.235551    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49beff55-c21a-477c-bfed-566df02ff578-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm\" (UID: \"49beff55-c21a-477c-bfed-566df02ff578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"
Oct 31 19:00:00.235622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.235590    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v4cx\" (UniqueName: \"kubernetes.io/projected/49beff55-c21a-477c-bfed-566df02ff578-kube-api-access-7v4cx\") pod \"maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm\" (UID: \"49beff55-c21a-477c-bfed-566df02ff578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"
Oct 31 19:00:00.261379 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.261338    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"]
Oct 31 19:00:00.264289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.264268    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"]
Oct 31 19:00:00.309824 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.309790    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"]
Oct 31 19:00:00.314123 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.313786    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"]
Oct 31 19:00:00.323089 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.316993    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:00:00.323089 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.320283    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"]
Oct 31 19:00:00.326716 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.326674    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"]
Oct 31 19:00:00.337149 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.337118    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55018466-4443-4d03-a113-c05f8c96980f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks\" (UID: \"55018466-4443-4d03-a113-c05f8c96980f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"
Oct 31 19:00:00.337297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.337153    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh5rh\" (UniqueName: \"kubernetes.io/projected/5f10ee1e-021c-45ea-b301-a50639c9731d-kube-api-access-gh5rh\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq\" (UID: \"5f10ee1e-021c-45ea-b301-a50639c9731d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"
Oct 31 19:00:00.337297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.337186    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h86d\" (UniqueName: \"kubernetes.io/projected/03a88726-34b2-42a5-901d-7e667a5518b1-kube-api-access-7h86d\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg\" (UID: \"03a88726-34b2-42a5-901d-7e667a5518b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"
Oct 31 19:00:00.337297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.337210    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49beff55-c21a-477c-bfed-566df02ff578-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm\" (UID: \"49beff55-c21a-477c-bfed-566df02ff578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"
Oct 31 19:00:00.337297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.337230    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4cx\" (UniqueName: \"kubernetes.io/projected/49beff55-c21a-477c-bfed-566df02ff578-kube-api-access-7v4cx\") pod \"maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm\" (UID: \"49beff55-c21a-477c-bfed-566df02ff578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"
Oct 31 19:00:00.337926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.337907    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49beff55-c21a-477c-bfed-566df02ff578-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm\" (UID: \"49beff55-c21a-477c-bfed-566df02ff578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"
Oct 31 19:00:00.338006 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.337945    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55018466-4443-4d03-a113-c05f8c96980f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks\" (UID: \"55018466-4443-4d03-a113-c05f8c96980f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"
Oct 31 19:00:00.338006 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.337973    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzt9c\" (UniqueName: \"kubernetes.io/projected/f55f4860-4889-4dc2-b8da-bce5bf87bfef-kube-api-access-bzt9c\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k\" (UID: \"f55f4860-4889-4dc2-b8da-bce5bf87bfef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"
Oct 31 19:00:00.338106 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.338070    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnd8v\" (UniqueName: \"kubernetes.io/projected/b557b692-500f-408d-945e-c5b688f142ee-kube-api-access-pnd8v\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7\" (UID: \"b557b692-500f-408d-945e-c5b688f142ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"
Oct 31 19:00:00.338170 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.338155    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55f4860-4889-4dc2-b8da-bce5bf87bfef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k\" (UID: \"f55f4860-4889-4dc2-b8da-bce5bf87bfef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"
Oct 31 19:00:00.338226 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.338213    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a88726-34b2-42a5-901d-7e667a5518b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg\" (UID: \"03a88726-34b2-42a5-901d-7e667a5518b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"
Oct 31 19:00:00.338282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.338273    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b557b692-500f-408d-945e-c5b688f142ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7\" (UID: \"b557b692-500f-408d-945e-c5b688f142ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"
Oct 31 19:00:00.338344 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.338326    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7t27\" (UniqueName: \"kubernetes.io/projected/55018466-4443-4d03-a113-c05f8c96980f-kube-api-access-n7t27\") pod \"maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks\" (UID: \"55018466-4443-4d03-a113-c05f8c96980f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"
Oct 31 19:00:00.338401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.338348    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f10ee1e-021c-45ea-b301-a50639c9731d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq\" (UID: \"5f10ee1e-021c-45ea-b301-a50639c9731d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"
Oct 31 19:00:00.338901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.338885    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f10ee1e-021c-45ea-b301-a50639c9731d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq\" (UID: \"5f10ee1e-021c-45ea-b301-a50639c9731d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"
Oct 31 19:00:00.394966 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.394936    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v4cx\" (UniqueName: \"kubernetes.io/projected/49beff55-c21a-477c-bfed-566df02ff578-kube-api-access-7v4cx\") pod \"maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm\" (UID: \"49beff55-c21a-477c-bfed-566df02ff578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"
Oct 31 19:00:00.395533 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.395510    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh5rh\" (UniqueName: \"kubernetes.io/projected/5f10ee1e-021c-45ea-b301-a50639c9731d-kube-api-access-gh5rh\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq\" (UID: \"5f10ee1e-021c-45ea-b301-a50639c9731d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"
Oct 31 19:00:00.395842 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.395823    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7t27\" (UniqueName: \"kubernetes.io/projected/55018466-4443-4d03-a113-c05f8c96980f-kube-api-access-n7t27\") pod \"maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks\" (UID: \"55018466-4443-4d03-a113-c05f8c96980f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"
Oct 31 19:00:00.439355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.439291    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h86d\" (UniqueName: \"kubernetes.io/projected/03a88726-34b2-42a5-901d-7e667a5518b1-kube-api-access-7h86d\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg\" (UID: \"03a88726-34b2-42a5-901d-7e667a5518b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"
Oct 31 19:00:00.439355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.439337    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzt9c\" (UniqueName: \"kubernetes.io/projected/f55f4860-4889-4dc2-b8da-bce5bf87bfef-kube-api-access-bzt9c\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k\" (UID: \"f55f4860-4889-4dc2-b8da-bce5bf87bfef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"
Oct 31 19:00:00.439567 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.439460    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnd8v\" (UniqueName: \"kubernetes.io/projected/b557b692-500f-408d-945e-c5b688f142ee-kube-api-access-pnd8v\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7\" (UID: \"b557b692-500f-408d-945e-c5b688f142ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"
Oct 31 19:00:00.439606 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.439565    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55f4860-4889-4dc2-b8da-bce5bf87bfef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k\" (UID: \"f55f4860-4889-4dc2-b8da-bce5bf87bfef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"
Oct 31 19:00:00.439643 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.439620    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a88726-34b2-42a5-901d-7e667a5518b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg\" (UID: \"03a88726-34b2-42a5-901d-7e667a5518b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"
Oct 31 19:00:00.439705 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.439689    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b557b692-500f-408d-945e-c5b688f142ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7\" (UID: \"b557b692-500f-408d-945e-c5b688f142ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"
Oct 31 19:00:00.440191 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.440176    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55f4860-4889-4dc2-b8da-bce5bf87bfef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k\" (UID: \"f55f4860-4889-4dc2-b8da-bce5bf87bfef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"
Oct 31 19:00:00.440234 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.440186    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a88726-34b2-42a5-901d-7e667a5518b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg\" (UID: \"03a88726-34b2-42a5-901d-7e667a5518b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"
Oct 31 19:00:00.440234 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.440226    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b557b692-500f-408d-945e-c5b688f142ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7\" (UID: \"b557b692-500f-408d-945e-c5b688f142ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"
Oct 31 19:00:00.481743 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.481716    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"
Oct 31 19:00:00.487978 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.487962    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"
Oct 31 19:00:00.497370 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.497341    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzt9c\" (UniqueName: \"kubernetes.io/projected/f55f4860-4889-4dc2-b8da-bce5bf87bfef-kube-api-access-bzt9c\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k\" (UID: \"f55f4860-4889-4dc2-b8da-bce5bf87bfef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"
Oct 31 19:00:00.497544 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.497525    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnd8v\" (UniqueName: \"kubernetes.io/projected/b557b692-500f-408d-945e-c5b688f142ee-kube-api-access-pnd8v\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7\" (UID: \"b557b692-500f-408d-945e-c5b688f142ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"
Oct 31 19:00:00.497595 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.497526    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h86d\" (UniqueName: \"kubernetes.io/projected/03a88726-34b2-42a5-901d-7e667a5518b1-kube-api-access-7h86d\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg\" (UID: \"03a88726-34b2-42a5-901d-7e667a5518b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"
Oct 31 19:00:00.503602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.503578    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"
Oct 31 19:00:00.530607 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.529660    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"
Oct 31 19:00:00.537191 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.536988    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"
Oct 31 19:00:00.555375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.555350    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"
Oct 31 19:00:00.874871 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.874740    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"]
Oct 31 19:00:00.918713 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.918688    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"]
Oct 31 19:00:00.918913 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:00:00.918885    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49beff55_c21a_477c_bfed_566df02ff578.slice/crio-1f318f30ed355799b46a767841d85608e82128a1904abc7357ec5c7def7be62d WatchSource:0}: Error finding container 1f318f30ed355799b46a767841d85608e82128a1904abc7357ec5c7def7be62d: Status 404 returned error can't find the container with id 1f318f30ed355799b46a767841d85608e82128a1904abc7357ec5c7def7be62d
Oct 31 19:00:00.931039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.931018    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"]
Oct 31 19:00:00.931216 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:00:00.931194    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f10ee1e_021c_45ea_b301_a50639c9731d.slice/crio-7610abb65b06810069a4c95ac12f3b696557191047aec352609ca5e242c64051 WatchSource:0}: Error finding container 7610abb65b06810069a4c95ac12f3b696557191047aec352609ca5e242c64051: Status 404 returned error can't find the container with id 7610abb65b06810069a4c95ac12f3b696557191047aec352609ca5e242c64051
Oct 31 19:00:00.971441 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.971421    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"]
Oct 31 19:00:00.971870 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:00:00.971847    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55f4860_4889_4dc2_b8da_bce5bf87bfef.slice/crio-8c59abbdec068142e231d643f16384ec4ddae5f06257dff615b47c833c8a330d WatchSource:0}: Error finding container 8c59abbdec068142e231d643f16384ec4ddae5f06257dff615b47c833c8a330d: Status 404 returned error can't find the container with id 8c59abbdec068142e231d643f16384ec4ddae5f06257dff615b47c833c8a330d
Oct 31 19:00:00.988083 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.987915    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"]
Oct 31 19:00:00.997931 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:00.997903    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"]
Oct 31 19:00:01.818811 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.818764    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7" event={"ID":"b557b692-500f-408d-945e-c5b688f142ee","Type":"ContainerStarted","Data":"e9780d7fbbc909e32e8723245d142a95ae75e0de92da11e6dea00ccf5af7d99b"}
Oct 31 19:00:01.819267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.818834    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7" event={"ID":"b557b692-500f-408d-945e-c5b688f142ee","Type":"ContainerStarted","Data":"69465d933f5e32dfe9f4914a316aa11dce1c5b97a696644470970cd659f8f50e"}
Oct 31 19:00:01.820349 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.820326    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq" event={"ID":"5f10ee1e-021c-45ea-b301-a50639c9731d","Type":"ContainerStarted","Data":"f09c53013ef5f42d4e393c16ac020c55613c7ade5a9a80f084af46f2fb14db9a"}
Oct 31 19:00:01.820461 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.820372    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq" event={"ID":"5f10ee1e-021c-45ea-b301-a50639c9731d","Type":"ContainerStarted","Data":"7610abb65b06810069a4c95ac12f3b696557191047aec352609ca5e242c64051"}
Oct 31 19:00:01.821792 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.821771    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg" event={"ID":"03a88726-34b2-42a5-901d-7e667a5518b1","Type":"ContainerStarted","Data":"30709413b760b776e60f0531cc3e6b7ce1d83a8b7d67d80163007c98b398b5d9"}
Oct 31 19:00:01.821869 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.821806    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg" event={"ID":"03a88726-34b2-42a5-901d-7e667a5518b1","Type":"ContainerStarted","Data":"16cb98040b488a1d664760eea5978ea85fb087b3b44cb70453f27f909e253315"}
Oct 31 19:00:01.823312 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.823293    4484 generic.go:356] "Generic (PLEG): container finished" podID="49beff55-c21a-477c-bfed-566df02ff578" containerID="e4c955a417253cd0918056f834606508f80e50fd3fc2279581d9fa351a36dbd3" exitCode=60
Oct 31 19:00:01.823400 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.823368    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm" event={"ID":"49beff55-c21a-477c-bfed-566df02ff578","Type":"ContainerDied","Data":"e4c955a417253cd0918056f834606508f80e50fd3fc2279581d9fa351a36dbd3"}
Oct 31 19:00:01.823446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.823409    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm" event={"ID":"49beff55-c21a-477c-bfed-566df02ff578","Type":"ContainerStarted","Data":"1f318f30ed355799b46a767841d85608e82128a1904abc7357ec5c7def7be62d"}
Oct 31 19:00:01.823552 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.823537    4484 scope.go:122] "RemoveContainer" containerID="e4c955a417253cd0918056f834606508f80e50fd3fc2279581d9fa351a36dbd3"
Oct 31 19:00:01.824965 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.824946    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks" event={"ID":"55018466-4443-4d03-a113-c05f8c96980f","Type":"ContainerStarted","Data":"efea288493014e33488859aac875ecce471ce945d36be4da9b3bccd43d6bb622"}
Oct 31 19:00:01.825032 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.824987    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks" event={"ID":"55018466-4443-4d03-a113-c05f8c96980f","Type":"ContainerStarted","Data":"aa2bd76a7eaeded70a6d05608804b0a83d625852dc52c5b25d3e3397d47c3ea1"}
Oct 31 19:00:01.826424 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.826320    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k" event={"ID":"f55f4860-4889-4dc2-b8da-bce5bf87bfef","Type":"ContainerStarted","Data":"f8b119bfe965a989afdc0a01da73827f735ac91d5019adb043c09059c80df182"}
Oct 31 19:00:01.826424 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.826349    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k" event={"ID":"f55f4860-4889-4dc2-b8da-bce5bf87bfef","Type":"ContainerStarted","Data":"8c59abbdec068142e231d643f16384ec4ddae5f06257dff615b47c833c8a330d"}
Oct 31 19:00:01.935040 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:01.934987    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7" podStartSLOduration=1.934974188 podStartE2EDuration="1.934974188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:01.932080273 +0000 UTC m=+2348.078530137" watchObservedRunningTime="2026-10-31 19:00:01.934974188 +0000 UTC m=+2348.081424052"
Oct 31 19:00:02.046522 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.046400    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k" podStartSLOduration=2.04638697 podStartE2EDuration="2.04638697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:02.044771506 +0000 UTC m=+2348.191221371" watchObservedRunningTime="2026-10-31 19:00:02.04638697 +0000 UTC m=+2348.192836844"
Oct 31 19:00:02.270739 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.270685    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg" podStartSLOduration=2.270669966 podStartE2EDuration="2.270669966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:02.269384713 +0000 UTC m=+2348.415834576" watchObservedRunningTime="2026-10-31 19:00:02.270669966 +0000 UTC m=+2348.417119877"
Oct 31 19:00:02.384929 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.384881    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks" podStartSLOduration=2.384866608 podStartE2EDuration="2.384866608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:02.382784234 +0000 UTC m=+2348.529234099" watchObservedRunningTime="2026-10-31 19:00:02.384866608 +0000 UTC m=+2348.531316472"
Oct 31 19:00:02.498082 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.498030    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq" podStartSLOduration=2.498017704 podStartE2EDuration="2.498017704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:02.497028145 +0000 UTC m=+2348.643478009" watchObservedRunningTime="2026-10-31 19:00:02.498017704 +0000 UTC m=+2348.644467567"
Oct 31 19:00:02.835497 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.835441    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm" event={"ID":"49beff55-c21a-477c-bfed-566df02ff578","Type":"ContainerStarted","Data":"08228afe38a20b0981b78439f6584c4d3cee96f44a918468801d6bbe5edaf9b9"}
Oct 31 19:00:02.836948 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.836924    4484 generic.go:356] "Generic (PLEG): container finished" podID="55018466-4443-4d03-a113-c05f8c96980f" containerID="efea288493014e33488859aac875ecce471ce945d36be4da9b3bccd43d6bb622" exitCode=60
Oct 31 19:00:02.837062 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.837001    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks" event={"ID":"55018466-4443-4d03-a113-c05f8c96980f","Type":"ContainerDied","Data":"efea288493014e33488859aac875ecce471ce945d36be4da9b3bccd43d6bb622"}
Oct 31 19:00:02.837205 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.837187    4484 scope.go:122] "RemoveContainer" containerID="efea288493014e33488859aac875ecce471ce945d36be4da9b3bccd43d6bb622"
Oct 31 19:00:02.838645 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.838564    4484 generic.go:356] "Generic (PLEG): container finished" podID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" containerID="f8b119bfe965a989afdc0a01da73827f735ac91d5019adb043c09059c80df182" exitCode=60
Oct 31 19:00:02.838709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.838662    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k" event={"ID":"f55f4860-4889-4dc2-b8da-bce5bf87bfef","Type":"ContainerDied","Data":"f8b119bfe965a989afdc0a01da73827f735ac91d5019adb043c09059c80df182"}
Oct 31 19:00:02.838812 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.838792    4484 scope.go:122] "RemoveContainer" containerID="f8b119bfe965a989afdc0a01da73827f735ac91d5019adb043c09059c80df182"
Oct 31 19:00:02.840352 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.840216    4484 generic.go:356] "Generic (PLEG): container finished" podID="b557b692-500f-408d-945e-c5b688f142ee" containerID="e9780d7fbbc909e32e8723245d142a95ae75e0de92da11e6dea00ccf5af7d99b" exitCode=60
Oct 31 19:00:02.840352 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.840297    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7" event={"ID":"b557b692-500f-408d-945e-c5b688f142ee","Type":"ContainerDied","Data":"e9780d7fbbc909e32e8723245d142a95ae75e0de92da11e6dea00ccf5af7d99b"}
Oct 31 19:00:02.840500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.840445    4484 scope.go:122] "RemoveContainer" containerID="e9780d7fbbc909e32e8723245d142a95ae75e0de92da11e6dea00ccf5af7d99b"
Oct 31 19:00:02.841963 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.841948    4484 generic.go:356] "Generic (PLEG): container finished" podID="5f10ee1e-021c-45ea-b301-a50639c9731d" containerID="f09c53013ef5f42d4e393c16ac020c55613c7ade5a9a80f084af46f2fb14db9a" exitCode=60
Oct 31 19:00:02.842076 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.842024    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq" event={"ID":"5f10ee1e-021c-45ea-b301-a50639c9731d","Type":"ContainerDied","Data":"f09c53013ef5f42d4e393c16ac020c55613c7ade5a9a80f084af46f2fb14db9a"}
Oct 31 19:00:02.842181 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.842162    4484 scope.go:122] "RemoveContainer" containerID="f09c53013ef5f42d4e393c16ac020c55613c7ade5a9a80f084af46f2fb14db9a"
Oct 31 19:00:02.843620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.843600    4484 generic.go:356] "Generic (PLEG): container finished" podID="03a88726-34b2-42a5-901d-7e667a5518b1" containerID="30709413b760b776e60f0531cc3e6b7ce1d83a8b7d67d80163007c98b398b5d9" exitCode=60
Oct 31 19:00:02.843728 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.843623    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg" event={"ID":"03a88726-34b2-42a5-901d-7e667a5518b1","Type":"ContainerDied","Data":"30709413b760b776e60f0531cc3e6b7ce1d83a8b7d67d80163007c98b398b5d9"}
Oct 31 19:00:02.843858 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.843841    4484 scope.go:122] "RemoveContainer" containerID="30709413b760b776e60f0531cc3e6b7ce1d83a8b7d67d80163007c98b398b5d9"
Oct 31 19:00:02.949963 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:02.949923    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm" podStartSLOduration=2.94991008 podStartE2EDuration="2.94991008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:02.948516985 +0000 UTC m=+2349.094966848" watchObservedRunningTime="2026-10-31 19:00:02.94991008 +0000 UTC m=+2349.096359942"
Oct 31 19:00:03.851947 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:03.851915    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg" event={"ID":"03a88726-34b2-42a5-901d-7e667a5518b1","Type":"ContainerStarted","Data":"4e28c3c2099a12c9ed64080249a38e0b800c9d2f51f5c9a7e6dab7678da7acf9"}
Oct 31 19:00:03.853875 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:03.853852    4484 generic.go:356] "Generic (PLEG): container finished" podID="49beff55-c21a-477c-bfed-566df02ff578" containerID="08228afe38a20b0981b78439f6584c4d3cee96f44a918468801d6bbe5edaf9b9" exitCode=60
Oct 31 19:00:03.853994 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:03.853930    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm" event={"ID":"49beff55-c21a-477c-bfed-566df02ff578","Type":"ContainerDied","Data":"08228afe38a20b0981b78439f6584c4d3cee96f44a918468801d6bbe5edaf9b9"}
Oct 31 19:00:03.853994 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:03.853979    4484 scope.go:122] "RemoveContainer" containerID="e4c955a417253cd0918056f834606508f80e50fd3fc2279581d9fa351a36dbd3"
Oct 31 19:00:03.854141 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:03.854125    4484 scope.go:122] "RemoveContainer" containerID="08228afe38a20b0981b78439f6584c4d3cee96f44a918468801d6bbe5edaf9b9"
Oct 31 19:00:03.854421 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:03.854389    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm_odh-ai-gateway-infra(49beff55-c21a-477c-bfed-566df02ff578)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm" podUID="49beff55-c21a-477c-bfed-566df02ff578"
Oct 31 19:00:03.856146 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:03.856123    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks" event={"ID":"55018466-4443-4d03-a113-c05f8c96980f","Type":"ContainerStarted","Data":"1c3db6a6e22108a1f5e9f978415afaa1b66039fdb9f1a00767e40be646834cb8"}
Oct 31 19:00:03.858082 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:03.858061    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k" event={"ID":"f55f4860-4889-4dc2-b8da-bce5bf87bfef","Type":"ContainerStarted","Data":"a294409ecdc7bef514251884aa46b4bc7c6eabf57220554eb92f167ba81d4815"}
Oct 31 19:00:03.859714 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:03.859693    4484 generic.go:356] "Generic (PLEG): container finished" podID="b557b692-500f-408d-945e-c5b688f142ee" containerID="c06c9505995a6c89eee43183636d54bb2057faf98fe804007dd892ad37f3bfb1" exitCode=60
Oct 31 19:00:03.859815 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:03.859771    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7" event={"ID":"b557b692-500f-408d-945e-c5b688f142ee","Type":"ContainerDied","Data":"c06c9505995a6c89eee43183636d54bb2057faf98fe804007dd892ad37f3bfb1"}
Oct 31 19:00:03.860040 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:03.860023    4484 scope.go:122] "RemoveContainer" containerID="c06c9505995a6c89eee43183636d54bb2057faf98fe804007dd892ad37f3bfb1"
Oct 31 19:00:03.860276 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:03.860251    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7_odh-ai-gateway-infra(b557b692-500f-408d-945e-c5b688f142ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7" podUID="b557b692-500f-408d-945e-c5b688f142ee"
Oct 31 19:00:03.870502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:03.862446    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq" event={"ID":"5f10ee1e-021c-45ea-b301-a50639c9731d","Type":"ContainerStarted","Data":"e7290b26122a4aebd871cf01ed67cc3289c6b66956dd3702b01680d859a87b92"}
Oct 31 19:00:03.924097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:03.924062    4484 scope.go:122] "RemoveContainer" containerID="e9780d7fbbc909e32e8723245d142a95ae75e0de92da11e6dea00ccf5af7d99b"
Oct 31 19:00:04.132263 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:04.132224    4484 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a88726_34b2_42a5_901d_7e667a5518b1.slice/crio-4e28c3c2099a12c9ed64080249a38e0b800c9d2f51f5c9a7e6dab7678da7acf9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a88726_34b2_42a5_901d_7e667a5518b1.slice/crio-conmon-4e28c3c2099a12c9ed64080249a38e0b800c9d2f51f5c9a7e6dab7678da7acf9.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 19:00:04.302540 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.302446    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:04.870328 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.870297    4484 scope.go:122] "RemoveContainer" containerID="08228afe38a20b0981b78439f6584c4d3cee96f44a918468801d6bbe5edaf9b9"
Oct 31 19:00:04.870784 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:04.870609    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm_odh-ai-gateway-infra(49beff55-c21a-477c-bfed-566df02ff578)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm" podUID="49beff55-c21a-477c-bfed-566df02ff578"
Oct 31 19:00:04.871809 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.871786    4484 generic.go:356] "Generic (PLEG): container finished" podID="55018466-4443-4d03-a113-c05f8c96980f" containerID="1c3db6a6e22108a1f5e9f978415afaa1b66039fdb9f1a00767e40be646834cb8" exitCode=60
Oct 31 19:00:04.871933 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.871816    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks" event={"ID":"55018466-4443-4d03-a113-c05f8c96980f","Type":"ContainerDied","Data":"1c3db6a6e22108a1f5e9f978415afaa1b66039fdb9f1a00767e40be646834cb8"}
Oct 31 19:00:04.871933 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.871877    4484 scope.go:122] "RemoveContainer" containerID="efea288493014e33488859aac875ecce471ce945d36be4da9b3bccd43d6bb622"
Oct 31 19:00:04.872143 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.872124    4484 scope.go:122] "RemoveContainer" containerID="1c3db6a6e22108a1f5e9f978415afaa1b66039fdb9f1a00767e40be646834cb8"
Oct 31 19:00:04.872427 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:04.872403    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks_odh-ai-gateway-infra(55018466-4443-4d03-a113-c05f8c96980f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks" podUID="55018466-4443-4d03-a113-c05f8c96980f"
Oct 31 19:00:04.873769 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.873751    4484 generic.go:356] "Generic (PLEG): container finished" podID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" containerID="a294409ecdc7bef514251884aa46b4bc7c6eabf57220554eb92f167ba81d4815" exitCode=60
Oct 31 19:00:04.873849 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.873826    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k" event={"ID":"f55f4860-4889-4dc2-b8da-bce5bf87bfef","Type":"ContainerDied","Data":"a294409ecdc7bef514251884aa46b4bc7c6eabf57220554eb92f167ba81d4815"}
Oct 31 19:00:04.873998 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.873981    4484 scope.go:122] "RemoveContainer" containerID="a294409ecdc7bef514251884aa46b4bc7c6eabf57220554eb92f167ba81d4815"
Oct 31 19:00:04.874207 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:04.874182    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k_odh-ai-gateway-infra(f55f4860-4889-4dc2-b8da-bce5bf87bfef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef"
Oct 31 19:00:04.875447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.875432    4484 scope.go:122] "RemoveContainer" containerID="c06c9505995a6c89eee43183636d54bb2057faf98fe804007dd892ad37f3bfb1"
Oct 31 19:00:04.875697 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:04.875676    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7_odh-ai-gateway-infra(b557b692-500f-408d-945e-c5b688f142ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7" podUID="b557b692-500f-408d-945e-c5b688f142ee"
Oct 31 19:00:04.876951 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.876933    4484 generic.go:356] "Generic (PLEG): container finished" podID="5f10ee1e-021c-45ea-b301-a50639c9731d" containerID="e7290b26122a4aebd871cf01ed67cc3289c6b66956dd3702b01680d859a87b92" exitCode=60
Oct 31 19:00:04.877053 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.877010    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq" event={"ID":"5f10ee1e-021c-45ea-b301-a50639c9731d","Type":"ContainerDied","Data":"e7290b26122a4aebd871cf01ed67cc3289c6b66956dd3702b01680d859a87b92"}
Oct 31 19:00:04.877178 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.877161    4484 scope.go:122] "RemoveContainer" containerID="e7290b26122a4aebd871cf01ed67cc3289c6b66956dd3702b01680d859a87b92"
Oct 31 19:00:04.877378 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:04.877360    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq_odh-ai-gateway-infra(5f10ee1e-021c-45ea-b301-a50639c9731d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d"
Oct 31 19:00:04.878916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.878899    4484 generic.go:356] "Generic (PLEG): container finished" podID="03a88726-34b2-42a5-901d-7e667a5518b1" containerID="4e28c3c2099a12c9ed64080249a38e0b800c9d2f51f5c9a7e6dab7678da7acf9" exitCode=60
Oct 31 19:00:04.878994 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.878944    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg" event={"ID":"03a88726-34b2-42a5-901d-7e667a5518b1","Type":"ContainerDied","Data":"4e28c3c2099a12c9ed64080249a38e0b800c9d2f51f5c9a7e6dab7678da7acf9"}
Oct 31 19:00:04.879170 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.879157    4484 scope.go:122] "RemoveContainer" containerID="4e28c3c2099a12c9ed64080249a38e0b800c9d2f51f5c9a7e6dab7678da7acf9"
Oct 31 19:00:04.879343 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:04.879329    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg_odh-ai-gateway-infra(03a88726-34b2-42a5-901d-7e667a5518b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg" podUID="03a88726-34b2-42a5-901d-7e667a5518b1"
Oct 31 19:00:04.884274 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.884257    4484 scope.go:122] "RemoveContainer" containerID="f8b119bfe965a989afdc0a01da73827f735ac91d5019adb043c09059c80df182"
Oct 31 19:00:04.896324 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.896306    4484 scope.go:122] "RemoveContainer" containerID="f09c53013ef5f42d4e393c16ac020c55613c7ade5a9a80f084af46f2fb14db9a"
Oct 31 19:00:04.908113 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:04.908098    4484 scope.go:122] "RemoveContainer" containerID="30709413b760b776e60f0531cc3e6b7ce1d83a8b7d67d80163007c98b398b5d9"
Oct 31 19:00:05.890942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:05.890910    4484 scope.go:122] "RemoveContainer" containerID="e7290b26122a4aebd871cf01ed67cc3289c6b66956dd3702b01680d859a87b92"
Oct 31 19:00:05.891359 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:05.891220    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq_odh-ai-gateway-infra(5f10ee1e-021c-45ea-b301-a50639c9731d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d"
Oct 31 19:00:05.892560 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:05.892539    4484 scope.go:122] "RemoveContainer" containerID="4e28c3c2099a12c9ed64080249a38e0b800c9d2f51f5c9a7e6dab7678da7acf9"
Oct 31 19:00:05.892800 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:05.892781    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg_odh-ai-gateway-infra(03a88726-34b2-42a5-901d-7e667a5518b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg" podUID="03a88726-34b2-42a5-901d-7e667a5518b1"
Oct 31 19:00:05.894109 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:05.894091    4484 scope.go:122] "RemoveContainer" containerID="1c3db6a6e22108a1f5e9f978415afaa1b66039fdb9f1a00767e40be646834cb8"
Oct 31 19:00:05.894347 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:05.894321    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks_odh-ai-gateway-infra(55018466-4443-4d03-a113-c05f8c96980f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks" podUID="55018466-4443-4d03-a113-c05f8c96980f"
Oct 31 19:00:05.895503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:05.895468    4484 scope.go:122] "RemoveContainer" containerID="a294409ecdc7bef514251884aa46b4bc7c6eabf57220554eb92f167ba81d4815"
Oct 31 19:00:05.895697 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:05.895682    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k_odh-ai-gateway-infra(f55f4860-4889-4dc2-b8da-bce5bf87bfef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef"
Oct 31 19:00:10.532313 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:10.532280    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:10.702060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:10.702028    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:10.855484 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:10.855375    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:14.986911 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:14.986877    4484 scope.go:122] "RemoveContainer" containerID="1c3db6a6e22108a1f5e9f978415afaa1b66039fdb9f1a00767e40be646834cb8"
Oct 31 19:00:15.488033 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:15.487997    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:15.956969 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:15.956937    4484 generic.go:356] "Generic (PLEG): container finished" podID="55018466-4443-4d03-a113-c05f8c96980f" containerID="d8e65d22c29143ada8c7b01e7a08433ad7d97c82d530ee61c10a26309f144aca" exitCode=60
Oct 31 19:00:15.957115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:15.957008    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks" event={"ID":"55018466-4443-4d03-a113-c05f8c96980f","Type":"ContainerDied","Data":"d8e65d22c29143ada8c7b01e7a08433ad7d97c82d530ee61c10a26309f144aca"}
Oct 31 19:00:15.957115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:15.957073    4484 scope.go:122] "RemoveContainer" containerID="1c3db6a6e22108a1f5e9f978415afaa1b66039fdb9f1a00767e40be646834cb8"
Oct 31 19:00:15.957426 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:15.957398    4484 scope.go:122] "RemoveContainer" containerID="d8e65d22c29143ada8c7b01e7a08433ad7d97c82d530ee61c10a26309f144aca"
Oct 31 19:00:15.957705 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:15.957687    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks_odh-ai-gateway-infra(55018466-4443-4d03-a113-c05f8c96980f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks" podUID="55018466-4443-4d03-a113-c05f8c96980f"
Oct 31 19:00:15.986663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:15.986648    4484 scope.go:122] "RemoveContainer" containerID="e7290b26122a4aebd871cf01ed67cc3289c6b66956dd3702b01680d859a87b92"
Oct 31 19:00:15.986752 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:15.986703    4484 scope.go:122] "RemoveContainer" containerID="4e28c3c2099a12c9ed64080249a38e0b800c9d2f51f5c9a7e6dab7678da7acf9"
Oct 31 19:00:16.964646 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:16.964622    4484 generic.go:356] "Generic (PLEG): container finished" podID="03a88726-34b2-42a5-901d-7e667a5518b1" containerID="341630cded0cd2cbaff330aeec4c8762dbf6884b6f8ec0fccf388b51719578a2" exitCode=60
Oct 31 19:00:16.965008 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:16.964699    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg" event={"ID":"03a88726-34b2-42a5-901d-7e667a5518b1","Type":"ContainerDied","Data":"341630cded0cd2cbaff330aeec4c8762dbf6884b6f8ec0fccf388b51719578a2"}
Oct 31 19:00:16.965008 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:16.964779    4484 scope.go:122] "RemoveContainer" containerID="4e28c3c2099a12c9ed64080249a38e0b800c9d2f51f5c9a7e6dab7678da7acf9"
Oct 31 19:00:16.965144 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:16.965121    4484 scope.go:122] "RemoveContainer" containerID="341630cded0cd2cbaff330aeec4c8762dbf6884b6f8ec0fccf388b51719578a2"
Oct 31 19:00:16.965439 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:16.965418    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg_odh-ai-gateway-infra(03a88726-34b2-42a5-901d-7e667a5518b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg" podUID="03a88726-34b2-42a5-901d-7e667a5518b1"
Oct 31 19:00:16.968766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:16.968744    4484 generic.go:356] "Generic (PLEG): container finished" podID="5f10ee1e-021c-45ea-b301-a50639c9731d" containerID="d4236c1fd271786d0c07183f7a9ca950d3692dc74efb6b9c7e4f3cef39e8cae4" exitCode=60
Oct 31 19:00:16.968852 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:16.968792    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq" event={"ID":"5f10ee1e-021c-45ea-b301-a50639c9731d","Type":"ContainerDied","Data":"d4236c1fd271786d0c07183f7a9ca950d3692dc74efb6b9c7e4f3cef39e8cae4"}
Oct 31 19:00:16.969114 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:16.969101    4484 scope.go:122] "RemoveContainer" containerID="d4236c1fd271786d0c07183f7a9ca950d3692dc74efb6b9c7e4f3cef39e8cae4"
Oct 31 19:00:16.969353 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:16.969334    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq_odh-ai-gateway-infra(5f10ee1e-021c-45ea-b301-a50639c9731d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d"
Oct 31 19:00:16.986619 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:16.986603    4484 scope.go:122] "RemoveContainer" containerID="a294409ecdc7bef514251884aa46b4bc7c6eabf57220554eb92f167ba81d4815"
Oct 31 19:00:16.986753 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:16.986735    4484 scope.go:122] "RemoveContainer" containerID="08228afe38a20b0981b78439f6584c4d3cee96f44a918468801d6bbe5edaf9b9"
Oct 31 19:00:16.987763 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:16.987741    4484 scope.go:122] "RemoveContainer" containerID="e7290b26122a4aebd871cf01ed67cc3289c6b66956dd3702b01680d859a87b92"
Oct 31 19:00:17.086468 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.086436    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"]
Oct 31 19:00:17.235735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.235712    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"
Oct 31 19:00:17.338557 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.338466    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55018466-4443-4d03-a113-c05f8c96980f-kube-api-access-n7t27\" (UniqueName: \"kubernetes.io/projected/55018466-4443-4d03-a113-c05f8c96980f-kube-api-access-n7t27\") pod \"55018466-4443-4d03-a113-c05f8c96980f\" (UID: \"55018466-4443-4d03-a113-c05f8c96980f\") "
Oct 31 19:00:17.338672 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.338608    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55018466-4443-4d03-a113-c05f8c96980f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55018466-4443-4d03-a113-c05f8c96980f-maas-api-service-ca\") pod \"55018466-4443-4d03-a113-c05f8c96980f\" (UID: \"55018466-4443-4d03-a113-c05f8c96980f\") "
Oct 31 19:00:17.338952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.338927    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55018466-4443-4d03-a113-c05f8c96980f-maas-api-service-ca" pod "55018466-4443-4d03-a113-c05f8c96980f" (UID: "55018466-4443-4d03-a113-c05f8c96980f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:00:17.340604 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.340583    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55018466-4443-4d03-a113-c05f8c96980f-kube-api-access-n7t27" pod "55018466-4443-4d03-a113-c05f8c96980f" (UID: "55018466-4443-4d03-a113-c05f8c96980f"). InnerVolumeSpecName "kube-api-access-n7t27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:17.440197 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.440164    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7t27\" (UniqueName: \"kubernetes.io/projected/55018466-4443-4d03-a113-c05f8c96980f-kube-api-access-n7t27\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:17.440197 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.440196    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55018466-4443-4d03-a113-c05f8c96980f-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:17.977906 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.977880    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"
Oct 31 19:00:17.978362 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.977886    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks" event={"ID":"55018466-4443-4d03-a113-c05f8c96980f","Type":"ContainerDied","Data":"aa2bd76a7eaeded70a6d05608804b0a83d625852dc52c5b25d3e3397d47c3ea1"}
Oct 31 19:00:17.978362 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.978029    4484 scope.go:122] "RemoveContainer" containerID="d8e65d22c29143ada8c7b01e7a08433ad7d97c82d530ee61c10a26309f144aca"
Oct 31 19:00:17.979973 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.979945    4484 generic.go:356] "Generic (PLEG): container finished" podID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" containerID="d121eb85cd092df74ac8fc7c7b9dba522f61507d1f54200595ef91c88114733c" exitCode=60
Oct 31 19:00:17.980115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.980018    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k" event={"ID":"f55f4860-4889-4dc2-b8da-bce5bf87bfef","Type":"ContainerDied","Data":"d121eb85cd092df74ac8fc7c7b9dba522f61507d1f54200595ef91c88114733c"}
Oct 31 19:00:17.980446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.980425    4484 scope.go:122] "RemoveContainer" containerID="d121eb85cd092df74ac8fc7c7b9dba522f61507d1f54200595ef91c88114733c"
Oct 31 19:00:17.980770 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:17.980745    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k_odh-ai-gateway-infra(f55f4860-4889-4dc2-b8da-bce5bf87bfef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef"
Oct 31 19:00:17.985679 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.985659    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm" event={"ID":"49beff55-c21a-477c-bfed-566df02ff578","Type":"ContainerStarted","Data":"ea0bc68b3634c0776c218d1bedd6af861b4585779998c44095a27491bbcc9ecc"}
Oct 31 19:00:17.986280 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.986264    4484 scope.go:122] "RemoveContainer" containerID="c06c9505995a6c89eee43183636d54bb2057faf98fe804007dd892ad37f3bfb1"
Oct 31 19:00:17.990080 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:17.989916    4484 scope.go:122] "RemoveContainer" containerID="a294409ecdc7bef514251884aa46b4bc7c6eabf57220554eb92f167ba81d4815"
Oct 31 19:00:18.091657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.091630    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"]
Oct 31 19:00:18.207930 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.207895    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"]
Oct 31 19:00:18.276585 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.276564    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"
Oct 31 19:00:18.350963 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.350935    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a88726-34b2-42a5-901d-7e667a5518b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a88726-34b2-42a5-901d-7e667a5518b1-maas-api-service-ca\") pod \"03a88726-34b2-42a5-901d-7e667a5518b1\" (UID: \"03a88726-34b2-42a5-901d-7e667a5518b1\") "
Oct 31 19:00:18.351151 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.351022    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a88726-34b2-42a5-901d-7e667a5518b1-kube-api-access-7h86d\" (UniqueName: \"kubernetes.io/projected/03a88726-34b2-42a5-901d-7e667a5518b1-kube-api-access-7h86d\") pod \"03a88726-34b2-42a5-901d-7e667a5518b1\" (UID: \"03a88726-34b2-42a5-901d-7e667a5518b1\") "
Oct 31 19:00:18.351573 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.351546    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a88726-34b2-42a5-901d-7e667a5518b1-maas-api-service-ca" pod "03a88726-34b2-42a5-901d-7e667a5518b1" (UID: "03a88726-34b2-42a5-901d-7e667a5518b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:00:18.353430 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.353408    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a88726-34b2-42a5-901d-7e667a5518b1-kube-api-access-7h86d" pod "03a88726-34b2-42a5-901d-7e667a5518b1" (UID: "03a88726-34b2-42a5-901d-7e667a5518b1"). InnerVolumeSpecName "kube-api-access-7h86d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:18.373641 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.373623    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"
Oct 31 19:00:18.381007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.380990    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"]
Oct 31 19:00:18.386269 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.386249    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-676224-29891220-9sxks"]
Oct 31 19:00:18.452900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.452878    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f10ee1e-021c-45ea-b301-a50639c9731d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f10ee1e-021c-45ea-b301-a50639c9731d-maas-api-service-ca\") pod \"5f10ee1e-021c-45ea-b301-a50639c9731d\" (UID: \"5f10ee1e-021c-45ea-b301-a50639c9731d\") "
Oct 31 19:00:18.453009 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.452956    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f10ee1e-021c-45ea-b301-a50639c9731d-kube-api-access-gh5rh\" (UniqueName: \"kubernetes.io/projected/5f10ee1e-021c-45ea-b301-a50639c9731d-kube-api-access-gh5rh\") pod \"5f10ee1e-021c-45ea-b301-a50639c9731d\" (UID: \"5f10ee1e-021c-45ea-b301-a50639c9731d\") "
Oct 31 19:00:18.453249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.453229    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f10ee1e-021c-45ea-b301-a50639c9731d-maas-api-service-ca" pod "5f10ee1e-021c-45ea-b301-a50639c9731d" (UID: "5f10ee1e-021c-45ea-b301-a50639c9731d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:00:18.453400 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.453385    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h86d\" (UniqueName: \"kubernetes.io/projected/03a88726-34b2-42a5-901d-7e667a5518b1-kube-api-access-7h86d\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:18.453488 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.453401    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f10ee1e-021c-45ea-b301-a50639c9731d-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:18.453488 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.453411    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03a88726-34b2-42a5-901d-7e667a5518b1-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:18.455115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.455095    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f10ee1e-021c-45ea-b301-a50639c9731d-kube-api-access-gh5rh" pod "5f10ee1e-021c-45ea-b301-a50639c9731d" (UID: "5f10ee1e-021c-45ea-b301-a50639c9731d"). InnerVolumeSpecName "kube-api-access-gh5rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:18.554383 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.554305    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh5rh\" (UniqueName: \"kubernetes.io/projected/5f10ee1e-021c-45ea-b301-a50639c9731d-kube-api-access-gh5rh\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:18.997407 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.997374    4484 generic.go:356] "Generic (PLEG): container finished" podID="b557b692-500f-408d-945e-c5b688f142ee" containerID="fa5af8371ca07621a384d0a9cfb857379f30b625fa478168d5e46cecf1c24473" exitCode=60
Oct 31 19:00:18.997923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.997446    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7" event={"ID":"b557b692-500f-408d-945e-c5b688f142ee","Type":"ContainerDied","Data":"fa5af8371ca07621a384d0a9cfb857379f30b625fa478168d5e46cecf1c24473"}
Oct 31 19:00:18.997923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.997531    4484 scope.go:122] "RemoveContainer" containerID="c06c9505995a6c89eee43183636d54bb2057faf98fe804007dd892ad37f3bfb1"
Oct 31 19:00:18.997923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.997895    4484 scope.go:122] "RemoveContainer" containerID="fa5af8371ca07621a384d0a9cfb857379f30b625fa478168d5e46cecf1c24473"
Oct 31 19:00:18.998191 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:18.998167    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7_odh-ai-gateway-infra(b557b692-500f-408d-945e-c5b688f142ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7" podUID="b557b692-500f-408d-945e-c5b688f142ee"
Oct 31 19:00:18.998901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.998839    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"
Oct 31 19:00:18.998901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:18.998878    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq" event={"ID":"5f10ee1e-021c-45ea-b301-a50639c9731d","Type":"ContainerDied","Data":"7610abb65b06810069a4c95ac12f3b696557191047aec352609ca5e242c64051"}
Oct 31 19:00:19.000224 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.000206    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"
Oct 31 19:00:19.000355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.000206    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg" event={"ID":"03a88726-34b2-42a5-901d-7e667a5518b1","Type":"ContainerDied","Data":"16cb98040b488a1d664760eea5978ea85fb087b3b44cb70453f27f909e253315"}
Oct 31 19:00:19.002226 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.002206    4484 generic.go:356] "Generic (PLEG): container finished" podID="49beff55-c21a-477c-bfed-566df02ff578" containerID="ea0bc68b3634c0776c218d1bedd6af861b4585779998c44095a27491bbcc9ecc" exitCode=60
Oct 31 19:00:19.002352 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.002272    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm" event={"ID":"49beff55-c21a-477c-bfed-566df02ff578","Type":"ContainerDied","Data":"ea0bc68b3634c0776c218d1bedd6af861b4585779998c44095a27491bbcc9ecc"}
Oct 31 19:00:19.002663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.002643    4484 scope.go:122] "RemoveContainer" containerID="ea0bc68b3634c0776c218d1bedd6af861b4585779998c44095a27491bbcc9ecc"
Oct 31 19:00:19.002945 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:19.002918    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm_odh-ai-gateway-infra(49beff55-c21a-477c-bfed-566df02ff578)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm" podUID="49beff55-c21a-477c-bfed-566df02ff578"
Oct 31 19:00:19.011787 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.011770    4484 scope.go:122] "RemoveContainer" containerID="d4236c1fd271786d0c07183f7a9ca950d3692dc74efb6b9c7e4f3cef39e8cae4"
Oct 31 19:00:19.020499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.020465    4484 scope.go:122] "RemoveContainer" containerID="341630cded0cd2cbaff330aeec4c8762dbf6884b6f8ec0fccf388b51719578a2"
Oct 31 19:00:19.041537 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.041517    4484 scope.go:122] "RemoveContainer" containerID="08228afe38a20b0981b78439f6584c4d3cee96f44a918468801d6bbe5edaf9b9"
Oct 31 19:00:19.155164 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.155095    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"]
Oct 31 19:00:19.274786 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.274757    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"]
Oct 31 19:00:19.318882 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.318851    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"
Oct 31 19:00:19.341556 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.341530    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"]
Oct 31 19:00:19.347105 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.347079    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891220-2hvcg"]
Oct 31 19:00:19.364916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.364892    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55f4860-4889-4dc2-b8da-bce5bf87bfef-kube-api-access-bzt9c\" (UniqueName: \"kubernetes.io/projected/f55f4860-4889-4dc2-b8da-bce5bf87bfef-kube-api-access-bzt9c\") pod \"f55f4860-4889-4dc2-b8da-bce5bf87bfef\" (UID: \"f55f4860-4889-4dc2-b8da-bce5bf87bfef\") "
Oct 31 19:00:19.365034 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.364941    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55f4860-4889-4dc2-b8da-bce5bf87bfef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55f4860-4889-4dc2-b8da-bce5bf87bfef-maas-api-service-ca\") pod \"f55f4860-4889-4dc2-b8da-bce5bf87bfef\" (UID: \"f55f4860-4889-4dc2-b8da-bce5bf87bfef\") "
Oct 31 19:00:19.365304 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.365276    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55f4860-4889-4dc2-b8da-bce5bf87bfef-maas-api-service-ca" pod "f55f4860-4889-4dc2-b8da-bce5bf87bfef" (UID: "f55f4860-4889-4dc2-b8da-bce5bf87bfef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:00:19.367038 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.367016    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55f4860-4889-4dc2-b8da-bce5bf87bfef-kube-api-access-bzt9c" pod "f55f4860-4889-4dc2-b8da-bce5bf87bfef" (UID: "f55f4860-4889-4dc2-b8da-bce5bf87bfef"). InnerVolumeSpecName "kube-api-access-bzt9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:19.466857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.466789    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzt9c\" (UniqueName: \"kubernetes.io/projected/f55f4860-4889-4dc2-b8da-bce5bf87bfef-kube-api-access-bzt9c\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:19.466857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.466818    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f55f4860-4889-4dc2-b8da-bce5bf87bfef-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:19.467140 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.467122    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"]
Oct 31 19:00:19.481198 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.481174    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891220-sxvfq"]
Oct 31 19:00:19.994546 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.994517    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a88726-34b2-42a5-901d-7e667a5518b1" path="/var/lib/kubelet/pods/03a88726-34b2-42a5-901d-7e667a5518b1/volumes"
Oct 31 19:00:19.994907 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.994895    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55018466-4443-4d03-a113-c05f8c96980f" path="/var/lib/kubelet/pods/55018466-4443-4d03-a113-c05f8c96980f/volumes"
Oct 31 19:00:19.995238 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:19.995227    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d" path="/var/lib/kubelet/pods/5f10ee1e-021c-45ea-b301-a50639c9731d/volumes"
Oct 31 19:00:20.016552 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.016524    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k" event={"ID":"f55f4860-4889-4dc2-b8da-bce5bf87bfef","Type":"ContainerDied","Data":"8c59abbdec068142e231d643f16384ec4ddae5f06257dff615b47c833c8a330d"}
Oct 31 19:00:20.016552 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.016541    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"
Oct 31 19:00:20.016982 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.016573    4484 scope.go:122] "RemoveContainer" containerID="d121eb85cd092df74ac8fc7c7b9dba522f61507d1f54200595ef91c88114733c"
Oct 31 19:00:20.123124 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.123093    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"]
Oct 31 19:00:20.180140 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.180112    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"
Oct 31 19:00:20.245433 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.245353    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"]
Oct 31 19:00:20.249698 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.249674    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891220-gmm4k"]
Oct 31 19:00:20.276864 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.276842    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49beff55-c21a-477c-bfed-566df02ff578-kube-api-access-7v4cx\" (UniqueName: \"kubernetes.io/projected/49beff55-c21a-477c-bfed-566df02ff578-kube-api-access-7v4cx\") pod \"49beff55-c21a-477c-bfed-566df02ff578\" (UID: \"49beff55-c21a-477c-bfed-566df02ff578\") "
Oct 31 19:00:20.276983 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.276888    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49beff55-c21a-477c-bfed-566df02ff578-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49beff55-c21a-477c-bfed-566df02ff578-maas-api-service-ca\") pod \"49beff55-c21a-477c-bfed-566df02ff578\" (UID: \"49beff55-c21a-477c-bfed-566df02ff578\") "
Oct 31 19:00:20.277252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.277227    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49beff55-c21a-477c-bfed-566df02ff578-maas-api-service-ca" pod "49beff55-c21a-477c-bfed-566df02ff578" (UID: "49beff55-c21a-477c-bfed-566df02ff578"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:00:20.278995 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.278973    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49beff55-c21a-477c-bfed-566df02ff578-kube-api-access-7v4cx" pod "49beff55-c21a-477c-bfed-566df02ff578" (UID: "49beff55-c21a-477c-bfed-566df02ff578"). InnerVolumeSpecName "kube-api-access-7v4cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:20.298002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.297985    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"
Oct 31 19:00:20.377682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.377652    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b557b692-500f-408d-945e-c5b688f142ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b557b692-500f-408d-945e-c5b688f142ee-maas-api-service-ca\") pod \"b557b692-500f-408d-945e-c5b688f142ee\" (UID: \"b557b692-500f-408d-945e-c5b688f142ee\") "
Oct 31 19:00:20.377820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.377708    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b557b692-500f-408d-945e-c5b688f142ee-kube-api-access-pnd8v\" (UniqueName: \"kubernetes.io/projected/b557b692-500f-408d-945e-c5b688f142ee-kube-api-access-pnd8v\") pod \"b557b692-500f-408d-945e-c5b688f142ee\" (UID: \"b557b692-500f-408d-945e-c5b688f142ee\") "
Oct 31 19:00:20.377974 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.377956    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b557b692-500f-408d-945e-c5b688f142ee-maas-api-service-ca" pod "b557b692-500f-408d-945e-c5b688f142ee" (UID: "b557b692-500f-408d-945e-c5b688f142ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:00:20.378130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.378104    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v4cx\" (UniqueName: \"kubernetes.io/projected/49beff55-c21a-477c-bfed-566df02ff578-kube-api-access-7v4cx\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:20.378130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.378125    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49beff55-c21a-477c-bfed-566df02ff578-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:20.378130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.378134    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b557b692-500f-408d-945e-c5b688f142ee-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:20.379913 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.379891    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b557b692-500f-408d-945e-c5b688f142ee-kube-api-access-pnd8v" pod "b557b692-500f-408d-945e-c5b688f142ee" (UID: "b557b692-500f-408d-945e-c5b688f142ee"). InnerVolumeSpecName "kube-api-access-pnd8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:20.479512 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.479464    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnd8v\" (UniqueName: \"kubernetes.io/projected/b557b692-500f-408d-945e-c5b688f142ee-kube-api-access-pnd8v\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:20.504369 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:20.504281    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:21.024880 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.024856    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"
Oct 31 19:00:21.024880 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.024868    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7" event={"ID":"b557b692-500f-408d-945e-c5b688f142ee","Type":"ContainerDied","Data":"69465d933f5e32dfe9f4914a316aa11dce1c5b97a696644470970cd659f8f50e"}
Oct 31 19:00:21.025404 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.024924    4484 scope.go:122] "RemoveContainer" containerID="fa5af8371ca07621a384d0a9cfb857379f30b625fa478168d5e46cecf1c24473"
Oct 31 19:00:21.026459 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.026423    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm" event={"ID":"49beff55-c21a-477c-bfed-566df02ff578","Type":"ContainerDied","Data":"1f318f30ed355799b46a767841d85608e82128a1904abc7357ec5c7def7be62d"}
Oct 31 19:00:21.026625 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.026431    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"
Oct 31 19:00:21.035059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.035045    4484 scope.go:122] "RemoveContainer" containerID="ea0bc68b3634c0776c218d1bedd6af861b4585779998c44095a27491bbcc9ecc"
Oct 31 19:00:21.194260 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.194236    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"]
Oct 31 19:00:21.196854 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.196831    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4157da-29891220-ht8vm"]
Oct 31 19:00:21.306740 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.306665    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"]
Oct 31 19:00:21.310467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.310445    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891220-v6tx7"]
Oct 31 19:00:21.994036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.994006    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49beff55-c21a-477c-bfed-566df02ff578" path="/var/lib/kubelet/pods/49beff55-c21a-477c-bfed-566df02ff578/volumes"
Oct 31 19:00:21.994413 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.994400    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b557b692-500f-408d-945e-c5b688f142ee" path="/var/lib/kubelet/pods/b557b692-500f-408d-945e-c5b688f142ee/volumes"
Oct 31 19:00:21.994783 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:21.994769    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" path="/var/lib/kubelet/pods/f55f4860-4889-4dc2-b8da-bce5bf87bfef/volumes"
Oct 31 19:00:24.003007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:24.002970    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:24.470315 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:24.470226    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:25.468219 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:25.468185    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:00:25.562935 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.562904    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"]
Oct 31 19:00:25.563551 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563532    4484 cpu_manager.go:414] "containerMap: removing container" podUID="55018466-4443-4d03-a113-c05f8c96980f" containerName="cleanup"
Oct 31 19:00:25.563551 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563551    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55018466-4443-4d03-a113-c05f8c96980f" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563565    4484 cpu_manager.go:414] "containerMap: removing container" podUID="b557b692-500f-408d-945e-c5b688f142ee" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563572    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b557b692-500f-408d-945e-c5b688f142ee" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563592    4484 cpu_manager.go:414] "containerMap: removing container" podUID="b557b692-500f-408d-945e-c5b688f142ee" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563600    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b557b692-500f-408d-945e-c5b688f142ee" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563629    4484 cpu_manager.go:414] "containerMap: removing container" podUID="49beff55-c21a-477c-bfed-566df02ff578" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563636    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49beff55-c21a-477c-bfed-566df02ff578" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563644    4484 cpu_manager.go:414] "containerMap: removing container" podUID="55018466-4443-4d03-a113-c05f8c96980f" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563651    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55018466-4443-4d03-a113-c05f8c96980f" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563661    4484 cpu_manager.go:414] "containerMap: removing container" podUID="03a88726-34b2-42a5-901d-7e667a5518b1" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563667    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a88726-34b2-42a5-901d-7e667a5518b1" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563688    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563695    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563711    4484 cpu_manager.go:414] "containerMap: removing container" podUID="03a88726-34b2-42a5-901d-7e667a5518b1" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563718    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a88726-34b2-42a5-901d-7e667a5518b1" containerName="cleanup"
Oct 31 19:00:25.563725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563727    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563734    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563743    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563750    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563763    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563770    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563783    4484 cpu_manager.go:414] "containerMap: removing container" podUID="49beff55-c21a-477c-bfed-566df02ff578" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563789    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49beff55-c21a-477c-bfed-566df02ff578" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563801    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563807    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563838    4484 cpu_manager.go:414] "containerMap: removing container" podUID="49beff55-c21a-477c-bfed-566df02ff578" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563845    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49beff55-c21a-477c-bfed-566df02ff578" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563855    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.563861    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564009    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="49beff55-c21a-477c-bfed-566df02ff578" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564020    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="55018466-4443-4d03-a113-c05f8c96980f" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564030    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a88726-34b2-42a5-901d-7e667a5518b1" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564042    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b557b692-500f-408d-945e-c5b688f142ee" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564052    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564060    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a88726-34b2-42a5-901d-7e667a5518b1" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564069    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a88726-34b2-42a5-901d-7e667a5518b1" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564080    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564087    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564098    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564108    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="49beff55-c21a-477c-bfed-566df02ff578" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564117    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55f4860-4889-4dc2-b8da-bce5bf87bfef" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564129    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="55018466-4443-4d03-a113-c05f8c96980f" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564136    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b557b692-500f-408d-945e-c5b688f142ee" containerName="cleanup"
Oct 31 19:00:25.564445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.564145    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b557b692-500f-408d-945e-c5b688f142ee" containerName="cleanup"
Oct 31 19:00:25.572332 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.572305    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"
Oct 31 19:00:25.621839 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.621809    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"]
Oct 31 19:00:25.628726 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.628699    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t76lq\" (UniqueName: \"kubernetes.io/projected/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-kube-api-access-t76lq\") pod \"payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74\" (UID: \"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"
Oct 31 19:00:25.628956 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.628935    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74\" (UID: \"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"
Oct 31 19:00:25.669177 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.669139    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-713bf0\""
Oct 31 19:00:25.669317 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.669222    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-713bf0-dockercfg-lnmd6\""
Oct 31 19:00:25.729831 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.729743    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74\" (UID: \"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"
Oct 31 19:00:25.730076 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.730048    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t76lq\" (UniqueName: \"kubernetes.io/projected/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-kube-api-access-t76lq\") pod \"payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74\" (UID: \"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"
Oct 31 19:00:25.730521 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.730502    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74\" (UID: \"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"
Oct 31 19:00:25.787777 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.787740    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t76lq\" (UniqueName: \"kubernetes.io/projected/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-kube-api-access-t76lq\") pod \"payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74\" (UID: \"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"
Oct 31 19:00:25.901451 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:25.901421    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"
Oct 31 19:00:26.277054 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:26.277023    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"]
Oct 31 19:00:26.277176 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:00:26.277117    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61eb1d02_c82b_4e2d_8751_a0f318f4e8e2.slice/crio-ee73b978db8656dc0e574033c51d007af39e01ead2cc9e5d83f3cecb1f4c96ad WatchSource:0}: Error finding container ee73b978db8656dc0e574033c51d007af39e01ead2cc9e5d83f3cecb1f4c96ad: Status 404 returned error can't find the container with id ee73b978db8656dc0e574033c51d007af39e01ead2cc9e5d83f3cecb1f4c96ad
Oct 31 19:00:26.966812 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:26.966777    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:27.071329 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:27.071297    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74" event={"ID":"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2","Type":"ContainerStarted","Data":"5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e"}
Oct 31 19:00:27.071551 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:27.071349    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74" event={"ID":"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2","Type":"ContainerStarted","Data":"ee73b978db8656dc0e574033c51d007af39e01ead2cc9e5d83f3cecb1f4c96ad"}
Oct 31 19:00:27.071551 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:27.071496    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"
Oct 31 19:00:27.187280 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:27.187237    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74" podStartSLOduration=2.187219512 podStartE2EDuration="2.187219512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:27.187074849 +0000 UTC m=+2373.333524713" watchObservedRunningTime="2026-10-31 19:00:27.187219512 +0000 UTC m=+2373.333669375"
Oct 31 19:00:28.934652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.934622    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"]
Oct 31 19:00:28.935266 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.935251    4484 cpu_manager.go:414] "containerMap: removing container" podUID="b557b692-500f-408d-945e-c5b688f142ee" containerName="cleanup"
Oct 31 19:00:28.935307 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.935267    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b557b692-500f-408d-945e-c5b688f142ee" containerName="cleanup"
Oct 31 19:00:28.935307 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.935293    4484 cpu_manager.go:414] "containerMap: removing container" podUID="55018466-4443-4d03-a113-c05f8c96980f" containerName="cleanup"
Oct 31 19:00:28.935307 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.935300    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55018466-4443-4d03-a113-c05f8c96980f" containerName="cleanup"
Oct 31 19:00:28.935442 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.935323    4484 cpu_manager.go:414] "containerMap: removing container" podUID="03a88726-34b2-42a5-901d-7e667a5518b1" containerName="cleanup"
Oct 31 19:00:28.935442 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.935328    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a88726-34b2-42a5-901d-7e667a5518b1" containerName="cleanup"
Oct 31 19:00:28.935442 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.935430    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="55018466-4443-4d03-a113-c05f8c96980f" containerName="cleanup"
Oct 31 19:00:28.935442 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.935439    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f10ee1e-021c-45ea-b301-a50639c9731d" containerName="cleanup"
Oct 31 19:00:28.935442 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.935444    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="49beff55-c21a-477c-bfed-566df02ff578" containerName="cleanup"
Oct 31 19:00:28.939021 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.939001    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:28.991882 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.991852    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:00:28.995811 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:28.995783    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"]
Oct 31 19:00:29.020057 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.020028    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:00:29.072759 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.072731    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxv67\" (UniqueName: \"kubernetes.io/projected/7c95c0b0-fde1-4af1-98b3-4ba00b603623-kube-api-access-lxv67\") pod \"maas-api-revoke-sub-d61cd41f-czd8l\" (UID: \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:29.072920 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.072798    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d61cd41f-czd8l\" (UID: \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:29.072920 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.072879    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d61cd41f-czd8l\" (UID: \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:29.097021 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.096980    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:29.174661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.174627    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d61cd41f-czd8l\" (UID: \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:29.174830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.174762    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxv67\" (UniqueName: \"kubernetes.io/projected/7c95c0b0-fde1-4af1-98b3-4ba00b603623-kube-api-access-lxv67\") pod \"maas-api-revoke-sub-d61cd41f-czd8l\" (UID: \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:29.174914 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.174829    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d61cd41f-czd8l\" (UID: \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:29.175428 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.175402    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d61cd41f-czd8l\" (UID: \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:29.177459 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.177436    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d61cd41f-czd8l\" (UID: \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:29.232110 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.232037    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxv67\" (UniqueName: \"kubernetes.io/projected/7c95c0b0-fde1-4af1-98b3-4ba00b603623-kube-api-access-lxv67\") pod \"maas-api-revoke-sub-d61cd41f-czd8l\" (UID: \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:29.252435 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.252415    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:29.499082 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.499054    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:29.644225 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:29.644198    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"]
Oct 31 19:00:29.645029 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:00:29.645007    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c95c0b0_fde1_4af1_98b3_4ba00b603623.slice/crio-fa2e8eb618d044279ac6019ad8cd6232da31794ab91448dc3ae76130c411a13e WatchSource:0}: Error finding container fa2e8eb618d044279ac6019ad8cd6232da31794ab91448dc3ae76130c411a13e: Status 404 returned error can't find the container with id fa2e8eb618d044279ac6019ad8cd6232da31794ab91448dc3ae76130c411a13e
Oct 31 19:00:30.089857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:30.089830    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l" event={"ID":"7c95c0b0-fde1-4af1-98b3-4ba00b603623","Type":"ContainerStarted","Data":"39eb70184ec1c75e84be7aa4d14efb26db99128faaee7bf286fc7d97b244788e"}
Oct 31 19:00:30.090226 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:30.089878    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l" event={"ID":"7c95c0b0-fde1-4af1-98b3-4ba00b603623","Type":"ContainerStarted","Data":"fa2e8eb618d044279ac6019ad8cd6232da31794ab91448dc3ae76130c411a13e"}
Oct 31 19:00:30.205976 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:30.205886    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l" podStartSLOduration=2.205872586 podStartE2EDuration="2.205872586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:00:30.203352211 +0000 UTC m=+2376.349802078" watchObservedRunningTime="2026-10-31 19:00:30.205872586 +0000 UTC m=+2376.352322447"
Oct 31 19:00:31.097426 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:31.097394    4484 generic.go:356] "Generic (PLEG): container finished" podID="7c95c0b0-fde1-4af1-98b3-4ba00b603623" containerID="39eb70184ec1c75e84be7aa4d14efb26db99128faaee7bf286fc7d97b244788e" exitCode=0
Oct 31 19:00:31.097812 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:31.097495    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l" event={"ID":"7c95c0b0-fde1-4af1-98b3-4ba00b603623","Type":"ContainerDied","Data":"39eb70184ec1c75e84be7aa4d14efb26db99128faaee7bf286fc7d97b244788e"}
Oct 31 19:00:31.700087 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:31.700054    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:00:31.762518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:31.762468    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:32.281843 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:32.281820    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:32.412809 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:32.412734    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-xks-ca\") pod \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\" (UID: \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\") "
Oct 31 19:00:32.412956 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:32.412855    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-openshift-ca\") pod \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\" (UID: \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\") "
Oct 31 19:00:32.412956 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:32.412930    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c95c0b0-fde1-4af1-98b3-4ba00b603623-kube-api-access-lxv67\" (UniqueName: \"kubernetes.io/projected/7c95c0b0-fde1-4af1-98b3-4ba00b603623-kube-api-access-lxv67\") pod \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\" (UID: \"7c95c0b0-fde1-4af1-98b3-4ba00b603623\") "
Oct 31 19:00:32.414130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:32.414100    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-openshift-ca" pod "7c95c0b0-fde1-4af1-98b3-4ba00b603623" (UID: "7c95c0b0-fde1-4af1-98b3-4ba00b603623"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:00:32.415364 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:32.415340    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-xks-ca" pod "7c95c0b0-fde1-4af1-98b3-4ba00b603623" (UID: "7c95c0b0-fde1-4af1-98b3-4ba00b603623"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:00:32.415651 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:32.415633    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c95c0b0-fde1-4af1-98b3-4ba00b603623-kube-api-access-lxv67" pod "7c95c0b0-fde1-4af1-98b3-4ba00b603623" (UID: "7c95c0b0-fde1-4af1-98b3-4ba00b603623"). InnerVolumeSpecName "kube-api-access-lxv67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:32.514623 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:32.514594    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:32.514623 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:32.514618    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxv67\" (UniqueName: \"kubernetes.io/projected/7c95c0b0-fde1-4af1-98b3-4ba00b603623-kube-api-access-lxv67\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:32.514623 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:32.514629    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c95c0b0-fde1-4af1-98b3-4ba00b603623-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:33.117021 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:33.116987    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"
Oct 31 19:00:33.117021 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:33.117015    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l" event={"ID":"7c95c0b0-fde1-4af1-98b3-4ba00b603623","Type":"ContainerDied","Data":"fa2e8eb618d044279ac6019ad8cd6232da31794ab91448dc3ae76130c411a13e"}
Oct 31 19:00:33.117224 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:33.117061    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa2e8eb618d044279ac6019ad8cd6232da31794ab91448dc3ae76130c411a13e"
Oct 31 19:00:35.135054 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:35.135020    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:36.547064 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:36.546984    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:38.078987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:38.078960    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"
Oct 31 19:00:39.382520 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:39.382468    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:39.979583 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:39.979548    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:40.171870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:40.171820    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:40.736153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:40.736114    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:42.255481 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:42.255443    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:43.188728 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:43.188694    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:43.347595 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:43.347563    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:44.648931 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:44.648892    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:45.181371 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:45.181329    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:45.351812 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:45.351778    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:45.519846 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:45.519808    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:46.644380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:46.644350    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:46.978349 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:46.978275    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"]
Oct 31 19:00:46.978533 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:46.978457    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp" podUID="16186434-2291-4ba4-bd98-ddcfa937eff6" containerName="payload-processing" containerID="cri-o://9240940e9856c7be2b4d16ab172c57f1d45b936cb2267e2fc3b318e56d4587b3" gracePeriod=30
Oct 31 19:00:47.161308 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:47.161255    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:47.209013 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:47.208785    4484 generic.go:356] "Generic (PLEG): container finished" podID="16186434-2291-4ba4-bd98-ddcfa937eff6" containerID="9240940e9856c7be2b4d16ab172c57f1d45b936cb2267e2fc3b318e56d4587b3" exitCode=0
Oct 31 19:00:47.209013 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:47.208935    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp" event={"ID":"16186434-2291-4ba4-bd98-ddcfa937eff6","Type":"ContainerDied","Data":"9240940e9856c7be2b4d16ab172c57f1d45b936cb2267e2fc3b318e56d4587b3"}
Oct 31 19:00:47.257693 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:47.257671    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"
Oct 31 19:00:47.273071 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:47.273047    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16186434-2291-4ba4-bd98-ddcfa937eff6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16186434-2291-4ba4-bd98-ddcfa937eff6-plugins-config-volume\") pod \"16186434-2291-4ba4-bd98-ddcfa937eff6\" (UID: \"16186434-2291-4ba4-bd98-ddcfa937eff6\") "
Oct 31 19:00:47.273224 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:47.273109    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16186434-2291-4ba4-bd98-ddcfa937eff6-kube-api-access-9l882\" (UniqueName: \"kubernetes.io/projected/16186434-2291-4ba4-bd98-ddcfa937eff6-kube-api-access-9l882\") pod \"16186434-2291-4ba4-bd98-ddcfa937eff6\" (UID: \"16186434-2291-4ba4-bd98-ddcfa937eff6\") "
Oct 31 19:00:47.273418 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:47.273395    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16186434-2291-4ba4-bd98-ddcfa937eff6-plugins-config-volume" pod "16186434-2291-4ba4-bd98-ddcfa937eff6" (UID: "16186434-2291-4ba4-bd98-ddcfa937eff6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:00:47.273554 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:47.273539    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16186434-2291-4ba4-bd98-ddcfa937eff6-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:47.275215 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:47.275192    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16186434-2291-4ba4-bd98-ddcfa937eff6-kube-api-access-9l882" pod "16186434-2291-4ba4-bd98-ddcfa937eff6" (UID: "16186434-2291-4ba4-bd98-ddcfa937eff6"). InnerVolumeSpecName "kube-api-access-9l882". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:00:47.321273 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:47.321245    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:47.375272 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:47.375244    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l882\" (UniqueName: \"kubernetes.io/projected/16186434-2291-4ba4-bd98-ddcfa937eff6-kube-api-access-9l882\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:00:48.216331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:48.216304    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"
Oct 31 19:00:48.216750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:48.216335    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp" event={"ID":"16186434-2291-4ba4-bd98-ddcfa937eff6","Type":"ContainerDied","Data":"585f11662ff0c1f2677e2ae8cbb063ec240197bd90acf090e9acc56a5d32f38e"}
Oct 31 19:00:48.216750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:48.216395    4484 scope.go:122] "RemoveContainer" containerID="9240940e9856c7be2b4d16ab172c57f1d45b936cb2267e2fc3b318e56d4587b3"
Oct 31 19:00:48.381263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:48.381168    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"]
Oct 31 19:00:48.431861 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:48.431833    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-676224-66fdfd85c6-p4hlp"]
Oct 31 19:00:48.469855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:48.469821    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:48.673157 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:48.673056    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:49.993940 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:49.993911    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16186434-2291-4ba4-bd98-ddcfa937eff6" path="/var/lib/kubelet/pods/16186434-2291-4ba4-bd98-ddcfa937eff6/volumes"
Oct 31 19:00:53.704242 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:53.704205    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:53.854551 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:53.854502    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:54.566396 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:54.566371    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:00:54.572514 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:54.572494    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:00:55.209611 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:55.209522    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:55.367611 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:55.367574    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:55.467834 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:00:55.467755    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:00:57.433425 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:57.433391    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:57.915689 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:57.915652    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:00:59.905925 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:00:59.905888    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:00.432945 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:00.432914    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:02.494988 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:02.494942    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:03.862902 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:03.862866    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:04.795678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:04.795641    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:08.116991 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:08.116957    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:08.307772 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:08.307733    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:08.669602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:08.669568    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"]
Oct 31 19:01:08.669822 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:08.669798    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs" podUID="fc20a9c6-8e4d-45ee-b309-39f286a35fe6" containerName="payload-pre-processing" containerID="cri-o://b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3" gracePeriod=30
Oct 31 19:01:08.788813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:08.788782    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:08.907419 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:08.907386    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"]
Oct 31 19:01:08.928536 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:08.928461    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"
Oct 31 19:01:08.936601 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:08.936571    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:09.015467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.015433    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-plugins-config-volume\") pod \"fc20a9c6-8e4d-45ee-b309-39f286a35fe6\" (UID: \"fc20a9c6-8e4d-45ee-b309-39f286a35fe6\") "
Oct 31 19:01:09.015641 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.015567    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-kube-api-access-jpsst\" (UniqueName: \"kubernetes.io/projected/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-kube-api-access-jpsst\") pod \"fc20a9c6-8e4d-45ee-b309-39f286a35fe6\" (UID: \"fc20a9c6-8e4d-45ee-b309-39f286a35fe6\") "
Oct 31 19:01:09.015909 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.015886    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"]
Oct 31 19:01:09.016702 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.016676    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-plugins-config-volume" pod "fc20a9c6-8e4d-45ee-b309-39f286a35fe6" (UID: "fc20a9c6-8e4d-45ee-b309-39f286a35fe6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:01:09.018263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.018228    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-kube-api-access-jpsst" pod "fc20a9c6-8e4d-45ee-b309-39f286a35fe6" (UID: "fc20a9c6-8e4d-45ee-b309-39f286a35fe6"). InnerVolumeSpecName "kube-api-access-jpsst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:01:09.117074 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.117043    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:09.117074 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.117078    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpsst\" (UniqueName: \"kubernetes.io/projected/fc20a9c6-8e4d-45ee-b309-39f286a35fe6-kube-api-access-jpsst\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:09.295892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.295859    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:09.356042 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.356011    4484 generic.go:356] "Generic (PLEG): container finished" podID="fc20a9c6-8e4d-45ee-b309-39f286a35fe6" containerID="b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3" exitCode=0
Oct 31 19:01:09.356219 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.356054    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"
Oct 31 19:01:09.356219 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.356083    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs" event={"ID":"fc20a9c6-8e4d-45ee-b309-39f286a35fe6","Type":"ContainerDied","Data":"b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3"}
Oct 31 19:01:09.356219 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.356135    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs" event={"ID":"fc20a9c6-8e4d-45ee-b309-39f286a35fe6","Type":"ContainerDied","Data":"335158a25cb3020275fa6c3ec301a50445a711672a527513114e64931bf47e8e"}
Oct 31 19:01:09.356219 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.356156    4484 scope.go:122] "RemoveContainer" containerID="b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3"
Oct 31 19:01:09.366558 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.366543    4484 scope.go:122] "RemoveContainer" containerID="b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3"
Oct 31 19:01:09.366813 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:01:09.366794    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3\": container with ID starting with b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3 not found: ID does not exist" containerID="b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3"
Oct 31 19:01:09.366860 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.366822    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3"} err="failed to get container status \"b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3\": rpc error: code = NotFound desc = could not find container \"b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3\": container with ID starting with b75cad3b30155f4e8f627fb82c2ed0fd1852e028b5509e5255178d60c0990ee3 not found: ID does not exist"
Oct 31 19:01:09.445034 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.444994    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:09.522118 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.522054    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"]
Oct 31 19:01:09.526259 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.526233    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-919b6e-6bd5f44fdc-nwtgs"]
Oct 31 19:01:09.994282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:09.994253    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc20a9c6-8e4d-45ee-b309-39f286a35fe6" path="/var/lib/kubelet/pods/fc20a9c6-8e4d-45ee-b309-39f286a35fe6/volumes"
Oct 31 19:01:10.242110 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:10.242073    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:10.431190 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:10.431094    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:10.592590 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:10.592554    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:11.978672 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:11.978629    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:12.473111 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:12.473080    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:13.908221 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:13.908182    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh" podUID="9c9e4a95-1f7c-423b-b6ff-1715f739359b" containerName="payload-processing" containerID="cri-o://2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6" gracePeriod=25
Oct 31 19:01:13.941024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:13.940998    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:14.017004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.016954    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg" podUID="460a6553-ba00-4794-b626-79da1137b729" containerName="payload-pre-processing" containerID="cri-o://066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0" gracePeriod=25
Oct 31 19:01:14.098145 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.090252    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:14.188250 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.188224    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"
Oct 31 19:01:14.295932 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.295902    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"
Oct 31 19:01:14.370527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.370467    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9e4a95-1f7c-423b-b6ff-1715f739359b-kube-api-access-qqndb\" (UniqueName: \"kubernetes.io/projected/9c9e4a95-1f7c-423b-b6ff-1715f739359b-kube-api-access-qqndb\") pod \"9c9e4a95-1f7c-423b-b6ff-1715f739359b\" (UID: \"9c9e4a95-1f7c-423b-b6ff-1715f739359b\") "
Oct 31 19:01:14.370692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.370533    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9e4a95-1f7c-423b-b6ff-1715f739359b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9e4a95-1f7c-423b-b6ff-1715f739359b-plugins-config-volume\") pod \"9c9e4a95-1f7c-423b-b6ff-1715f739359b\" (UID: \"9c9e4a95-1f7c-423b-b6ff-1715f739359b\") "
Oct 31 19:01:14.370914 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.370886    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9e4a95-1f7c-423b-b6ff-1715f739359b-plugins-config-volume" pod "9c9e4a95-1f7c-423b-b6ff-1715f739359b" (UID: "9c9e4a95-1f7c-423b-b6ff-1715f739359b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:01:14.372816 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.372796    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9e4a95-1f7c-423b-b6ff-1715f739359b-kube-api-access-qqndb" pod "9c9e4a95-1f7c-423b-b6ff-1715f739359b" (UID: "9c9e4a95-1f7c-423b-b6ff-1715f739359b"). InnerVolumeSpecName "kube-api-access-qqndb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:01:14.388329 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.388307    4484 generic.go:356] "Generic (PLEG): container finished" podID="9c9e4a95-1f7c-423b-b6ff-1715f739359b" containerID="2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6" exitCode=0
Oct 31 19:01:14.388412 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.388357    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"
Oct 31 19:01:14.388412 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.388396    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh" event={"ID":"9c9e4a95-1f7c-423b-b6ff-1715f739359b","Type":"ContainerDied","Data":"2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6"}
Oct 31 19:01:14.388522 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.388449    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh" event={"ID":"9c9e4a95-1f7c-423b-b6ff-1715f739359b","Type":"ContainerDied","Data":"7477c02b9eaf3268e611d67ac696f7771e2a3313855a87b26b911a4bf1fbcf97"}
Oct 31 19:01:14.388522 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.388506    4484 scope.go:122] "RemoveContainer" containerID="2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6"
Oct 31 19:01:14.389822 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.389796    4484 generic.go:356] "Generic (PLEG): container finished" podID="460a6553-ba00-4794-b626-79da1137b729" containerID="066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0" exitCode=0
Oct 31 19:01:14.389909 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.389823    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg" event={"ID":"460a6553-ba00-4794-b626-79da1137b729","Type":"ContainerDied","Data":"066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0"}
Oct 31 19:01:14.389909 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.389858    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"
Oct 31 19:01:14.389909 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.389868    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg" event={"ID":"460a6553-ba00-4794-b626-79da1137b729","Type":"ContainerDied","Data":"a9247bfc9cf7b5e90226773370158af1efd4d6fa1619fbee6587392d51978a61"}
Oct 31 19:01:14.398512 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.398468    4484 scope.go:122] "RemoveContainer" containerID="2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6"
Oct 31 19:01:14.398779 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:01:14.398762    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6\": container with ID starting with 2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6 not found: ID does not exist" containerID="2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6"
Oct 31 19:01:14.398865 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.398791    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6"} err="failed to get container status \"2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6\": rpc error: code = NotFound desc = could not find container \"2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6\": container with ID starting with 2a97217f225fd5d4388817fc06e1af4e836b5a1feeb5bda7bf657e41a2a1ebd6 not found: ID does not exist"
Oct 31 19:01:14.398865 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.398813    4484 scope.go:122] "RemoveContainer" containerID="066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0"
Oct 31 19:01:14.407384 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.407370    4484 scope.go:122] "RemoveContainer" containerID="066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0"
Oct 31 19:01:14.407635 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:01:14.407619    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0\": container with ID starting with 066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0 not found: ID does not exist" containerID="066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0"
Oct 31 19:01:14.407686 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.407640    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0"} err="failed to get container status \"066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0\": rpc error: code = NotFound desc = could not find container \"066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0\": container with ID starting with 066bdcd75822a49614db3f8ef4bb1a3188aa91df216f47be88a4b9168d7797e0 not found: ID does not exist"
Oct 31 19:01:14.471432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.471349    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/460a6553-ba00-4794-b626-79da1137b729-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460a6553-ba00-4794-b626-79da1137b729-plugins-config-volume\") pod \"460a6553-ba00-4794-b626-79da1137b729\" (UID: \"460a6553-ba00-4794-b626-79da1137b729\") "
Oct 31 19:01:14.471432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.471396    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/460a6553-ba00-4794-b626-79da1137b729-kube-api-access-c7plc\" (UniqueName: \"kubernetes.io/projected/460a6553-ba00-4794-b626-79da1137b729-kube-api-access-c7plc\") pod \"460a6553-ba00-4794-b626-79da1137b729\" (UID: \"460a6553-ba00-4794-b626-79da1137b729\") "
Oct 31 19:01:14.471839 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.471813    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460a6553-ba00-4794-b626-79da1137b729-plugins-config-volume" pod "460a6553-ba00-4794-b626-79da1137b729" (UID: "460a6553-ba00-4794-b626-79da1137b729"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:01:14.472011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.471994    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460a6553-ba00-4794-b626-79da1137b729-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:14.472267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.472025    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqndb\" (UniqueName: \"kubernetes.io/projected/9c9e4a95-1f7c-423b-b6ff-1715f739359b-kube-api-access-qqndb\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:14.472267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.472044    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9e4a95-1f7c-423b-b6ff-1715f739359b-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:14.474045 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.474020    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460a6553-ba00-4794-b626-79da1137b729-kube-api-access-c7plc" pod "460a6553-ba00-4794-b626-79da1137b729" (UID: "460a6553-ba00-4794-b626-79da1137b729"). InnerVolumeSpecName "kube-api-access-c7plc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:01:14.553802 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.553754    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:14.572967 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.572934    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7plc\" (UniqueName: \"kubernetes.io/projected/460a6553-ba00-4794-b626-79da1137b729-kube-api-access-c7plc\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:14.588456 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.588423    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"]
Oct 31 19:01:14.594440 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.594411    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4157da-765cd5f87f-4m6bh"]
Oct 31 19:01:14.754942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.754907    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:14.821325 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.821287    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"]
Oct 31 19:01:14.824914 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:14.824889    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4157da-68c9bd9675-fhxkg"]
Oct 31 19:01:15.427405 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:15.427373    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:15.590111 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:15.590067    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:15.993653 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:15.993624    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="460a6553-ba00-4794-b626-79da1137b729" path="/var/lib/kubelet/pods/460a6553-ba00-4794-b626-79da1137b729/volumes"
Oct 31 19:01:15.994023 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:15.994012    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9e4a95-1f7c-423b-b6ff-1715f739359b" path="/var/lib/kubelet/pods/9c9e4a95-1f7c-423b-b6ff-1715f739359b/volumes"
Oct 31 19:01:18.364898 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:18.364864    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:18.537520 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:18.537466    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:18.711239 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:18.711159    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:20.665596 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:20.665559    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:20.911027 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:20.910990    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:21.637467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:21.637435    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:01:23.044182 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.044149    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:23.213561 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.213527    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:23.591315 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.591279    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"]
Oct 31 19:01:23.591838 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.591825    4484 cpu_manager.go:414] "containerMap: removing container" podUID="7c95c0b0-fde1-4af1-98b3-4ba00b603623" containerName="revoke-keys"
Oct 31 19:01:23.591891 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.591838    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c95c0b0-fde1-4af1-98b3-4ba00b603623" containerName="revoke-keys"
Oct 31 19:01:23.591891 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.591856    4484 cpu_manager.go:414] "containerMap: removing container" podUID="460a6553-ba00-4794-b626-79da1137b729" containerName="payload-pre-processing"
Oct 31 19:01:23.591891 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.591861    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460a6553-ba00-4794-b626-79da1137b729" containerName="payload-pre-processing"
Oct 31 19:01:23.591891 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.591874    4484 cpu_manager.go:414] "containerMap: removing container" podUID="fc20a9c6-8e4d-45ee-b309-39f286a35fe6" containerName="payload-pre-processing"
Oct 31 19:01:23.591891 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.591880    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc20a9c6-8e4d-45ee-b309-39f286a35fe6" containerName="payload-pre-processing"
Oct 31 19:01:23.592036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.591906    4484 cpu_manager.go:414] "containerMap: removing container" podUID="9c9e4a95-1f7c-423b-b6ff-1715f739359b" containerName="payload-processing"
Oct 31 19:01:23.592036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.591911    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9e4a95-1f7c-423b-b6ff-1715f739359b" containerName="payload-processing"
Oct 31 19:01:23.592036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.591921    4484 cpu_manager.go:414] "containerMap: removing container" podUID="16186434-2291-4ba4-bd98-ddcfa937eff6" containerName="payload-processing"
Oct 31 19:01:23.592036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.591925    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16186434-2291-4ba4-bd98-ddcfa937eff6" containerName="payload-processing"
Oct 31 19:01:23.592036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.592002    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="16186434-2291-4ba4-bd98-ddcfa937eff6" containerName="payload-processing"
Oct 31 19:01:23.592036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.592012    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c95c0b0-fde1-4af1-98b3-4ba00b603623" containerName="revoke-keys"
Oct 31 19:01:23.592036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.592019    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="460a6553-ba00-4794-b626-79da1137b729" containerName="payload-pre-processing"
Oct 31 19:01:23.592036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.592025    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc20a9c6-8e4d-45ee-b309-39f286a35fe6" containerName="payload-pre-processing"
Oct 31 19:01:23.592036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.592036    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9e4a95-1f7c-423b-b6ff-1715f739359b" containerName="payload-processing"
Oct 31 19:01:23.595664 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.595647    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"
Oct 31 19:01:23.651922 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.651890    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"]
Oct 31 19:01:23.660261 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.660239    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3acf824f-751d-4b09-80ad-c1f88b785502-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d\" (UID: \"3acf824f-751d-4b09-80ad-c1f88b785502\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"
Oct 31 19:01:23.660379 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.660282    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpkxk\" (UniqueName: \"kubernetes.io/projected/3acf824f-751d-4b09-80ad-c1f88b785502-kube-api-access-gpkxk\") pod \"payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d\" (UID: \"3acf824f-751d-4b09-80ad-c1f88b785502\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"
Oct 31 19:01:23.698407 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.698379    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-79e7b6\""
Oct 31 19:01:23.698542 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.698501    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-79e7b6-dockercfg-5kjqb\""
Oct 31 19:01:23.761777 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.761747    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpkxk\" (UniqueName: \"kubernetes.io/projected/3acf824f-751d-4b09-80ad-c1f88b785502-kube-api-access-gpkxk\") pod \"payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d\" (UID: \"3acf824f-751d-4b09-80ad-c1f88b785502\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"
Oct 31 19:01:23.761914 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.761883    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3acf824f-751d-4b09-80ad-c1f88b785502-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d\" (UID: \"3acf824f-751d-4b09-80ad-c1f88b785502\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"
Oct 31 19:01:23.762541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.762520    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3acf824f-751d-4b09-80ad-c1f88b785502-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d\" (UID: \"3acf824f-751d-4b09-80ad-c1f88b785502\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"
Oct 31 19:01:23.802630 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.802606    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:23.819432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.819405    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpkxk\" (UniqueName: \"kubernetes.io/projected/3acf824f-751d-4b09-80ad-c1f88b785502-kube-api-access-gpkxk\") pod \"payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d\" (UID: \"3acf824f-751d-4b09-80ad-c1f88b785502\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"
Oct 31 19:01:23.911671 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:23.911611    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"
Oct 31 19:01:24.283622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:24.283601    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"]
Oct 31 19:01:24.284567 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:01:24.284545    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acf824f_751d_4b09_80ad_c1f88b785502.slice/crio-677aa78102c56b47e03de18661e4376903cbbebaf67d4dd964fd877be88928c7 WatchSource:0}: Error finding container 677aa78102c56b47e03de18661e4376903cbbebaf67d4dd964fd877be88928c7: Status 404 returned error can't find the container with id 677aa78102c56b47e03de18661e4376903cbbebaf67d4dd964fd877be88928c7
Oct 31 19:01:24.450862 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:24.450825    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d" event={"ID":"3acf824f-751d-4b09-80ad-c1f88b785502","Type":"ContainerStarted","Data":"4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a"}
Oct 31 19:01:24.451016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:24.451001    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"
Oct 31 19:01:24.451061 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:24.451035    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d" event={"ID":"3acf824f-751d-4b09-80ad-c1f88b785502","Type":"ContainerStarted","Data":"677aa78102c56b47e03de18661e4376903cbbebaf67d4dd964fd877be88928c7"}
Oct 31 19:01:24.567490 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:24.567383    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d" podStartSLOduration=1.567370433 podStartE2EDuration="1.567370433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:01:24.564654558 +0000 UTC m=+2430.711104421" watchObservedRunningTime="2026-10-31 19:01:24.567370433 +0000 UTC m=+2430.713820351"
Oct 31 19:01:25.468208 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:01:25.468165    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:01:27.010592 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:27.010554    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:29.956103 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:29.956065    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:30.444986 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:30.444956    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"]
Oct 31 19:01:30.556907 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:30.556870    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"]
Oct 31 19:01:30.823636 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:30.823599    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:30.975530 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:30.975493    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:31.890418 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:31.890390    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:35.419156 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.419118    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:35.445359 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.445311    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb" podUID="48c5af01-fd8f-4f73-adbb-3ee549cd0ef6" containerName="payload-processing" containerID="cri-o://398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693" gracePeriod=25
Oct 31 19:01:35.459857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.459836    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"
Oct 31 19:01:35.557550 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.557517    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv" podUID="23a7a922-9383-470e-8bcd-dcfc03082b29" containerName="payload-pre-processing" containerID="cri-o://0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27" gracePeriod=25
Oct 31 19:01:35.743701 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.743672    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"
Oct 31 19:01:35.793091 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.793061    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-plugins-config-volume\") pod \"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6\" (UID: \"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6\") "
Oct 31 19:01:35.793274 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.793114    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-kube-api-access-ffqnf\" (UniqueName: \"kubernetes.io/projected/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-kube-api-access-ffqnf\") pod \"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6\" (UID: \"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6\") "
Oct 31 19:01:35.793519 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.793487    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-plugins-config-volume" pod "48c5af01-fd8f-4f73-adbb-3ee549cd0ef6" (UID: "48c5af01-fd8f-4f73-adbb-3ee549cd0ef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:01:35.796079 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.796045    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-kube-api-access-ffqnf" pod "48c5af01-fd8f-4f73-adbb-3ee549cd0ef6" (UID: "48c5af01-fd8f-4f73-adbb-3ee549cd0ef6"). InnerVolumeSpecName "kube-api-access-ffqnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:01:35.830000 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.829980    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"
Oct 31 19:01:35.894429 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.894404    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a7a922-9383-470e-8bcd-dcfc03082b29-kube-api-access-4xr8b\" (UniqueName: \"kubernetes.io/projected/23a7a922-9383-470e-8bcd-dcfc03082b29-kube-api-access-4xr8b\") pod \"23a7a922-9383-470e-8bcd-dcfc03082b29\" (UID: \"23a7a922-9383-470e-8bcd-dcfc03082b29\") "
Oct 31 19:01:35.894578 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.894530    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a7a922-9383-470e-8bcd-dcfc03082b29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a7a922-9383-470e-8bcd-dcfc03082b29-plugins-config-volume\") pod \"23a7a922-9383-470e-8bcd-dcfc03082b29\" (UID: \"23a7a922-9383-470e-8bcd-dcfc03082b29\") "
Oct 31 19:01:35.894938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.894916    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:35.894938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.894937    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffqnf\" (UniqueName: \"kubernetes.io/projected/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6-kube-api-access-ffqnf\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:35.895135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.894947    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a7a922-9383-470e-8bcd-dcfc03082b29-plugins-config-volume" pod "23a7a922-9383-470e-8bcd-dcfc03082b29" (UID: "23a7a922-9383-470e-8bcd-dcfc03082b29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:01:35.896621 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.896602    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a7a922-9383-470e-8bcd-dcfc03082b29-kube-api-access-4xr8b" pod "23a7a922-9383-470e-8bcd-dcfc03082b29" (UID: "23a7a922-9383-470e-8bcd-dcfc03082b29"). InnerVolumeSpecName "kube-api-access-4xr8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:01:35.995781 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.995752    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xr8b\" (UniqueName: \"kubernetes.io/projected/23a7a922-9383-470e-8bcd-dcfc03082b29-kube-api-access-4xr8b\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:35.995781 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:35.995781    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a7a922-9383-470e-8bcd-dcfc03082b29-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:01:36.524042 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.524006    4484 generic.go:356] "Generic (PLEG): container finished" podID="48c5af01-fd8f-4f73-adbb-3ee549cd0ef6" containerID="398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693" exitCode=0
Oct 31 19:01:36.524507 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.524059    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"
Oct 31 19:01:36.524507 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.524078    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb" event={"ID":"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6","Type":"ContainerDied","Data":"398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693"}
Oct 31 19:01:36.524507 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.524132    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb" event={"ID":"48c5af01-fd8f-4f73-adbb-3ee549cd0ef6","Type":"ContainerDied","Data":"2b7949e6aedb68ad9e7e8970f71a143a3268ebf76077108f09c607e9f854f953"}
Oct 31 19:01:36.524507 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.524164    4484 scope.go:122] "RemoveContainer" containerID="398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693"
Oct 31 19:01:36.525593 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.525529    4484 generic.go:356] "Generic (PLEG): container finished" podID="23a7a922-9383-470e-8bcd-dcfc03082b29" containerID="0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27" exitCode=0
Oct 31 19:01:36.525593 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.525557    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv" event={"ID":"23a7a922-9383-470e-8bcd-dcfc03082b29","Type":"ContainerDied","Data":"0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27"}
Oct 31 19:01:36.525593 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.525584    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"
Oct 31 19:01:36.525787 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.525610    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv" event={"ID":"23a7a922-9383-470e-8bcd-dcfc03082b29","Type":"ContainerDied","Data":"5dd0b938fb1163bd879e39d5be93700c7cd6a5261e2650a26b6c12f180bf3180"}
Oct 31 19:01:36.533894 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.533877    4484 scope.go:122] "RemoveContainer" containerID="398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693"
Oct 31 19:01:36.534159 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:01:36.534143    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693\": container with ID starting with 398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693 not found: ID does not exist" containerID="398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693"
Oct 31 19:01:36.534209 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.534165    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693"} err="failed to get container status \"398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693\": rpc error: code = NotFound desc = could not find container \"398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693\": container with ID starting with 398e67e378f42f4565f2d69835622e84b3833ea473eca214754b258eff9c2693 not found: ID does not exist"
Oct 31 19:01:36.534209 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.534183    4484 scope.go:122] "RemoveContainer" containerID="0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27"
Oct 31 19:01:36.542772 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.542756    4484 scope.go:122] "RemoveContainer" containerID="0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27"
Oct 31 19:01:36.543008 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:01:36.542992    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27\": container with ID starting with 0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27 not found: ID does not exist" containerID="0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27"
Oct 31 19:01:36.543052 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.543014    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27"} err="failed to get container status \"0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27\": rpc error: code = NotFound desc = could not find container \"0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27\": container with ID starting with 0fbed5b883c91d01741ae4f5281b9f6f310440fa56d6578191ab055548f14c27 not found: ID does not exist"
Oct 31 19:01:36.658251 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.658216    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:36.688713 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.688661    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"]
Oct 31 19:01:36.691791 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.691766    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3932ef-6d6d4d5dcf-59xrb"]
Oct 31 19:01:36.798437 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.798405    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"]
Oct 31 19:01:36.800918 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:36.800892    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3932ef-56fd64654c-cksnv"]
Oct 31 19:01:37.993290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:37.993259    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a7a922-9383-470e-8bcd-dcfc03082b29" path="/var/lib/kubelet/pods/23a7a922-9383-470e-8bcd-dcfc03082b29/volumes"
Oct 31 19:01:37.993687 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:37.993673    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c5af01-fd8f-4f73-adbb-3ee549cd0ef6" path="/var/lib/kubelet/pods/48c5af01-fd8f-4f73-adbb-3ee549cd0ef6/volumes"
Oct 31 19:01:40.736849 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:40.736820    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:42.327575 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:42.327535    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:42.529034 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:42.529000    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:42.677329 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:42.677240    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:46.475498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:46.475438    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:01:46.605974 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:46.605939    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:46.783829 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:46.783795    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:47.316814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:47.316774    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:48.485855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:48.485822    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:48.970456 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:48.970428    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:49.755399 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:49.755370    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:52.607524 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:52.607485    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:54.033643 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:54.033608    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:55.449262 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:55.449226    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:55.468916 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:01:55.468887    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:01:56.494990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:56.494954    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"]
Oct 31 19:01:56.600911 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:56.600881    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"]
Oct 31 19:01:56.694916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:56.694886    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:56.861210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:56.861117    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:57.015170 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:57.015129    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:58.466264 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:58.466230    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:58.629854 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:58.629820    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:58.790832 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:58.790795    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:01:59.972497 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:01:59.972449    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:00.462132 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:00.462097    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:00.892118 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:00.892087    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"]
Oct 31 19:02:00.892310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:00.892292    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz" podUID="e68c29f1-193e-40f4-bcb5-62e42a865c98" containerName="istio-proxy" containerID="cri-o://1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b" gracePeriod=30
Oct 31 19:02:01.222582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.222493    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:01.365789 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.365737    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:01.496036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.495999    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp" podUID="169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc" containerName="payload-processing" containerID="cri-o://11c043720de85c97383c8a0c355f5e4a5b54f5b01108b80e8570b224e6b21269" gracePeriod=25
Oct 31 19:02:01.601832 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.601784    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8" podUID="091ccab3-9990-4362-a441-a4d8a8782b6b" containerName="payload-pre-processing" containerID="cri-o://1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2" gracePeriod=25
Oct 31 19:02:01.681550 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.681521    4484 generic.go:356] "Generic (PLEG): container finished" podID="169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc" containerID="11c043720de85c97383c8a0c355f5e4a5b54f5b01108b80e8570b224e6b21269" exitCode=0
Oct 31 19:02:01.681712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.681591    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp" event={"ID":"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc","Type":"ContainerDied","Data":"11c043720de85c97383c8a0c355f5e4a5b54f5b01108b80e8570b224e6b21269"}
Oct 31 19:02:01.758943 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.758880    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"
Oct 31 19:02:01.864174 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.864149    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"
Oct 31 19:02:01.882556 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.882528    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-plugins-config-volume\") pod \"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc\" (UID: \"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc\") "
Oct 31 19:02:01.882715 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.882567    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-kube-api-access-cflzf\" (UniqueName: \"kubernetes.io/projected/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-kube-api-access-cflzf\") pod \"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc\" (UID: \"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc\") "
Oct 31 19:02:01.882961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.882939    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-plugins-config-volume" pod "169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc" (UID: "169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:02:01.883084 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.883070    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:01.884911 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.884885    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-kube-api-access-cflzf" pod "169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc" (UID: "169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc"). InnerVolumeSpecName "kube-api-access-cflzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:02:01.984592 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.984555    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091ccab3-9990-4362-a441-a4d8a8782b6b-kube-api-access-2gs2n\" (UniqueName: \"kubernetes.io/projected/091ccab3-9990-4362-a441-a4d8a8782b6b-kube-api-access-2gs2n\") pod \"091ccab3-9990-4362-a441-a4d8a8782b6b\" (UID: \"091ccab3-9990-4362-a441-a4d8a8782b6b\") "
Oct 31 19:02:01.984782 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.984648    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091ccab3-9990-4362-a441-a4d8a8782b6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091ccab3-9990-4362-a441-a4d8a8782b6b-plugins-config-volume\") pod \"091ccab3-9990-4362-a441-a4d8a8782b6b\" (UID: \"091ccab3-9990-4362-a441-a4d8a8782b6b\") "
Oct 31 19:02:01.985005 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.984983    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cflzf\" (UniqueName: \"kubernetes.io/projected/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc-kube-api-access-cflzf\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:01.985121 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.984999    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091ccab3-9990-4362-a441-a4d8a8782b6b-plugins-config-volume" pod "091ccab3-9990-4362-a441-a4d8a8782b6b" (UID: "091ccab3-9990-4362-a441-a4d8a8782b6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:02:01.986874 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:01.986851    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091ccab3-9990-4362-a441-a4d8a8782b6b-kube-api-access-2gs2n" pod "091ccab3-9990-4362-a441-a4d8a8782b6b" (UID: "091ccab3-9990-4362-a441-a4d8a8782b6b"). InnerVolumeSpecName "kube-api-access-2gs2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:02:02.086101 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.086052    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/091ccab3-9990-4362-a441-a4d8a8782b6b-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:02.086101 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.086088    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gs2n\" (UniqueName: \"kubernetes.io/projected/091ccab3-9990-4362-a441-a4d8a8782b6b-kube-api-access-2gs2n\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:02.689239 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.689203    4484 generic.go:356] "Generic (PLEG): container finished" podID="091ccab3-9990-4362-a441-a4d8a8782b6b" containerID="1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2" exitCode=0
Oct 31 19:02:02.689768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.689267    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"
Oct 31 19:02:02.689768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.689291    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8" event={"ID":"091ccab3-9990-4362-a441-a4d8a8782b6b","Type":"ContainerDied","Data":"1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2"}
Oct 31 19:02:02.689768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.689378    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8" event={"ID":"091ccab3-9990-4362-a441-a4d8a8782b6b","Type":"ContainerDied","Data":"75ef37bd265ad47a55dfadc2a05cdfda39df9acbc88c7f16e3d256360b6f2df7"}
Oct 31 19:02:02.689768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.689412    4484 scope.go:122] "RemoveContainer" containerID="1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2"
Oct 31 19:02:02.691056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.691019    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp" event={"ID":"169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc","Type":"ContainerDied","Data":"cabcddbc24e0f09475ea8bea12425eb5bf537ccf79e730f4b77a9d9428978098"}
Oct 31 19:02:02.691056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.691024    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"
Oct 31 19:02:02.699873 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.699851    4484 scope.go:122] "RemoveContainer" containerID="1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2"
Oct 31 19:02:02.700220 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:02:02.700197    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2\": container with ID starting with 1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2 not found: ID does not exist" containerID="1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2"
Oct 31 19:02:02.700303 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.700229    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2"} err="failed to get container status \"1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2\": rpc error: code = NotFound desc = could not find container \"1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2\": container with ID starting with 1447964facf3a7dd540342501ea68d2560e08ffb28f8a85faa0d5830befdfbf2 not found: ID does not exist"
Oct 31 19:02:02.700303 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.700247    4484 scope.go:122] "RemoveContainer" containerID="11c043720de85c97383c8a0c355f5e4a5b54f5b01108b80e8570b224e6b21269"
Oct 31 19:02:02.856187 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.856105    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"]
Oct 31 19:02:02.906246 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:02.906214    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e43ccf-85fdfc4d7c-jrvcp"]
Oct 31 19:02:03.016645 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:03.016613    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"]
Oct 31 19:02:03.023805 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:03.023772    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e43ccf-86c9bfd7c5-llxw8"]
Oct 31 19:02:03.892812 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:03.892771    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:03.997776 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:03.997737    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091ccab3-9990-4362-a441-a4d8a8782b6b" path="/var/lib/kubelet/pods/091ccab3-9990-4362-a441-a4d8a8782b6b/volumes"
Oct 31 19:02:03.998364 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:03.998342    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc" path="/var/lib/kubelet/pods/169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc/volumes"
Oct 31 19:02:06.150056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.150025    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 19:02:06.232432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.232341    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-kube-api-access-dkqr5\" (UniqueName: \"kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-kube-api-access-dkqr5\") pod \"e68c29f1-193e-40f4-bcb5-62e42a865c98\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") "
Oct 31 19:02:06.232432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.232387    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-podinfo\") pod \"e68c29f1-193e-40f4-bcb5-62e42a865c98\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") "
Oct 31 19:02:06.232432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.232418    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-data\") pod \"e68c29f1-193e-40f4-bcb5-62e42a865c98\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") "
Oct 31 19:02:06.232765 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.232455    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68c29f1-193e-40f4-bcb5-62e42a865c98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68c29f1-193e-40f4-bcb5-62e42a865c98-istiod-ca-cert\") pod \"e68c29f1-193e-40f4-bcb5-62e42a865c98\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") "
Oct 31 19:02:06.232765 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.232629    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-token\" (UniqueName: \"kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-token\") pod \"e68c29f1-193e-40f4-bcb5-62e42a865c98\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") "
Oct 31 19:02:06.232765 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.232684    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-envoy\") pod \"e68c29f1-193e-40f4-bcb5-62e42a865c98\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") "
Oct 31 19:02:06.232765 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.232712    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-socket\") pod \"e68c29f1-193e-40f4-bcb5-62e42a865c98\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") "
Oct 31 19:02:06.232765 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.232751    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-credential-socket\") pod \"e68c29f1-193e-40f4-bcb5-62e42a865c98\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") "
Oct 31 19:02:06.233025 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.232770    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-data" pod "e68c29f1-193e-40f4-bcb5-62e42a865c98" (UID: "e68c29f1-193e-40f4-bcb5-62e42a865c98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:02:06.233025 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.232801    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-certs\") pod \"e68c29f1-193e-40f4-bcb5-62e42a865c98\" (UID: \"e68c29f1-193e-40f4-bcb5-62e42a865c98\") "
Oct 31 19:02:06.233025 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.232977    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-socket" pod "e68c29f1-193e-40f4-bcb5-62e42a865c98" (UID: "e68c29f1-193e-40f4-bcb5-62e42a865c98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:02:06.233247 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.233176    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-credential-socket" pod "e68c29f1-193e-40f4-bcb5-62e42a865c98" (UID: "e68c29f1-193e-40f4-bcb5-62e42a865c98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:02:06.233247 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.233204    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-certs" pod "e68c29f1-193e-40f4-bcb5-62e42a865c98" (UID: "e68c29f1-193e-40f4-bcb5-62e42a865c98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:02:06.233466 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.233322    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.233466 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.233340    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.233466 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.233353    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.233466 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.233367    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.233466 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.233393    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68c29f1-193e-40f4-bcb5-62e42a865c98-istiod-ca-cert" pod "e68c29f1-193e-40f4-bcb5-62e42a865c98" (UID: "e68c29f1-193e-40f4-bcb5-62e42a865c98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:02:06.235076 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.235048    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-podinfo" pod "e68c29f1-193e-40f4-bcb5-62e42a865c98" (UID: "e68c29f1-193e-40f4-bcb5-62e42a865c98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:02:06.235346 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.235302    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-envoy" pod "e68c29f1-193e-40f4-bcb5-62e42a865c98" (UID: "e68c29f1-193e-40f4-bcb5-62e42a865c98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:02:06.235926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.235906    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-kube-api-access-dkqr5" pod "e68c29f1-193e-40f4-bcb5-62e42a865c98" (UID: "e68c29f1-193e-40f4-bcb5-62e42a865c98"). InnerVolumeSpecName "kube-api-access-dkqr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:02:06.236018 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.235926    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-token" pod "e68c29f1-193e-40f4-bcb5-62e42a865c98" (UID: "e68c29f1-193e-40f4-bcb5-62e42a865c98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:02:06.306591 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.306560    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"]
Oct 31 19:02:06.334439 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.334403    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.334439 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.334440    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkqr5\" (UniqueName: \"kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-kube-api-access-dkqr5\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.334713 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.334455    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.334713 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.334468    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68c29f1-193e-40f4-bcb5-62e42a865c98-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.334713 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.334561    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68c29f1-193e-40f4-bcb5-62e42a865c98-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:06.412860 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.412824    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"]
Oct 31 19:02:06.686866 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.686827    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:06.731750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.731717    4484 generic.go:356] "Generic (PLEG): container finished" podID="e68c29f1-193e-40f4-bcb5-62e42a865c98" containerID="1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b" exitCode=0
Oct 31 19:02:06.731923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.731780    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz" event={"ID":"e68c29f1-193e-40f4-bcb5-62e42a865c98","Type":"ContainerDied","Data":"1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b"}
Oct 31 19:02:06.731923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.731795    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"
Oct 31 19:02:06.731923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.731858    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz" event={"ID":"e68c29f1-193e-40f4-bcb5-62e42a865c98","Type":"ContainerDied","Data":"02b081022f61db44e953c4aef9049badd3fcc0c3bf49f3533c8abc60fd79f50a"}
Oct 31 19:02:06.731923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.731886    4484 scope.go:122] "RemoveContainer" containerID="1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b"
Oct 31 19:02:06.748055 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.747976    4484 scope.go:122] "RemoveContainer" containerID="1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b"
Oct 31 19:02:06.748394 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:02:06.748367    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b\": container with ID starting with 1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b not found: ID does not exist" containerID="1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b"
Oct 31 19:02:06.748531 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.748408    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b"} err="failed to get container status \"1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b\": rpc error: code = NotFound desc = could not find container \"1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b\": container with ID starting with 1844fbfd492978115497ac71e3a4bf1da38981e76de4b3610645157eb115a13b not found: ID does not exist"
Oct 31 19:02:06.835231 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.835183    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:06.898680 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.898641    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"]
Oct 31 19:02:06.902877 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:06.902845    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-ktlzz"]
Oct 31 19:02:07.416629 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:07.416462    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:07.994502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:07.994424    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68c29f1-193e-40f4-bcb5-62e42a865c98" path="/var/lib/kubelet/pods/e68c29f1-193e-40f4-bcb5-62e42a865c98/volumes"
Oct 31 19:02:08.873327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:08.873285    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:10.352515 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.352458    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"]
Oct 31 19:02:10.353058 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353044    4484 cpu_manager.go:414] "containerMap: removing container" podUID="23a7a922-9383-470e-8bcd-dcfc03082b29" containerName="payload-pre-processing"
Oct 31 19:02:10.353108 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353059    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a7a922-9383-470e-8bcd-dcfc03082b29" containerName="payload-pre-processing"
Oct 31 19:02:10.353108 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353074    4484 cpu_manager.go:414] "containerMap: removing container" podUID="091ccab3-9990-4362-a441-a4d8a8782b6b" containerName="payload-pre-processing"
Oct 31 19:02:10.353108 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353080    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091ccab3-9990-4362-a441-a4d8a8782b6b" containerName="payload-pre-processing"
Oct 31 19:02:10.353108 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353094    4484 cpu_manager.go:414] "containerMap: removing container" podUID="e68c29f1-193e-40f4-bcb5-62e42a865c98" containerName="istio-proxy"
Oct 31 19:02:10.353108 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353101    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68c29f1-193e-40f4-bcb5-62e42a865c98" containerName="istio-proxy"
Oct 31 19:02:10.353281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353114    4484 cpu_manager.go:414] "containerMap: removing container" podUID="48c5af01-fd8f-4f73-adbb-3ee549cd0ef6" containerName="payload-processing"
Oct 31 19:02:10.353281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353120    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c5af01-fd8f-4f73-adbb-3ee549cd0ef6" containerName="payload-processing"
Oct 31 19:02:10.353281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353130    4484 cpu_manager.go:414] "containerMap: removing container" podUID="169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc" containerName="payload-processing"
Oct 31 19:02:10.353281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353136    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc" containerName="payload-processing"
Oct 31 19:02:10.353281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353231    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a7a922-9383-470e-8bcd-dcfc03082b29" containerName="payload-pre-processing"
Oct 31 19:02:10.353281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353241    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="091ccab3-9990-4362-a441-a4d8a8782b6b" containerName="payload-pre-processing"
Oct 31 19:02:10.353281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353250    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68c29f1-193e-40f4-bcb5-62e42a865c98" containerName="istio-proxy"
Oct 31 19:02:10.353281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353258    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c5af01-fd8f-4f73-adbb-3ee549cd0ef6" containerName="payload-processing"
Oct 31 19:02:10.353281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.353264    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="169ef4d7-7ac6-4436-a7fe-f9f2a951d5fc" containerName="payload-processing"
Oct 31 19:02:10.356691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.356676    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.459743 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.459699    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4c58c5-openshift-default-dockercfg-54lm7\""
Oct 31 19:02:10.463844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.463814    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"]
Oct 31 19:02:10.478435 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.478396    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.478661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.478452    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.478661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.478566    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.478780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.478668    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.478780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.478732    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.478885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.478866    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6fk8\" (UniqueName: \"kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-kube-api-access-b6fk8\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.478926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.478917    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.479010 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.478995    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.479046 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.479024    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.580567 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.580529    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.580729 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.580593    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.580729 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.580653    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6fk8\" (UniqueName: \"kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-kube-api-access-b6fk8\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.580729 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.580705    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.580860 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.580757    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.580860 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.580789    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.580970 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.580874    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.580970 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.580905    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.580970 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.580941    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.580970 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.580954    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.581621 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.581573    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.581621 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.581616    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.581781 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.581665    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.581931 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.581907    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.583327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.583303    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.584136 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.584113    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.639173 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.639096    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.639335 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.639169    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6fk8\" (UniqueName: \"kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-kube-api-access-b6fk8\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:10.670134 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:10.670100    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:11.099408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.099380    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"]
Oct 31 19:02:11.137950 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:02:11.137917    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be5dd8d_6d5a_4ed4_a5aa_23094889d17a.slice/crio-b7d2de7bb32ca0a089e732a61b6ff63a2293d93b92d1e9e4889676e257661a26 WatchSource:0}: Error finding container b7d2de7bb32ca0a089e732a61b6ff63a2293d93b92d1e9e4889676e257661a26: Status 404 returned error can't find the container with id b7d2de7bb32ca0a089e732a61b6ff63a2293d93b92d1e9e4889676e257661a26
Oct 31 19:02:11.140337 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.140303    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:02:11.140426 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.140373    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:02:11.140426 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.140404    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:02:11.307724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.307683    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4" podUID="34bac501-dd5d-499e-80e4-e7016b536662" containerName="payload-processing" containerID="cri-o://c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54" gracePeriod=25
Oct 31 19:02:11.413965 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.413853    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht" podUID="8ed1f34a-89a9-4dca-a574-e602d5471ed3" containerName="payload-pre-processing" containerID="cri-o://512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455" gracePeriod=25
Oct 31 19:02:11.552039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.552010    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"
Oct 31 19:02:11.609640 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.609615    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34bac501-dd5d-499e-80e4-e7016b536662-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34bac501-dd5d-499e-80e4-e7016b536662-plugins-config-volume\") pod \"34bac501-dd5d-499e-80e4-e7016b536662\" (UID: \"34bac501-dd5d-499e-80e4-e7016b536662\") "
Oct 31 19:02:11.609824 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.609675    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34bac501-dd5d-499e-80e4-e7016b536662-kube-api-access-lvp74\" (UniqueName: \"kubernetes.io/projected/34bac501-dd5d-499e-80e4-e7016b536662-kube-api-access-lvp74\") pod \"34bac501-dd5d-499e-80e4-e7016b536662\" (UID: \"34bac501-dd5d-499e-80e4-e7016b536662\") "
Oct 31 19:02:11.610079 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.610054    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34bac501-dd5d-499e-80e4-e7016b536662-plugins-config-volume" pod "34bac501-dd5d-499e-80e4-e7016b536662" (UID: "34bac501-dd5d-499e-80e4-e7016b536662"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:02:11.613940 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.613914    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34bac501-dd5d-499e-80e4-e7016b536662-kube-api-access-lvp74" pod "34bac501-dd5d-499e-80e4-e7016b536662" (UID: "34bac501-dd5d-499e-80e4-e7016b536662"). InnerVolumeSpecName "kube-api-access-lvp74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:02:11.694904 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.694877    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"
Oct 31 19:02:11.711629 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.711587    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34bac501-dd5d-499e-80e4-e7016b536662-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:11.711629 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.711623    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvp74\" (UniqueName: \"kubernetes.io/projected/34bac501-dd5d-499e-80e4-e7016b536662-kube-api-access-lvp74\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:11.765152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.765114    4484 generic.go:356] "Generic (PLEG): container finished" podID="8ed1f34a-89a9-4dca-a574-e602d5471ed3" containerID="512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455" exitCode=0
Oct 31 19:02:11.765365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.765220    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"
Oct 31 19:02:11.765365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.765262    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht" event={"ID":"8ed1f34a-89a9-4dca-a574-e602d5471ed3","Type":"ContainerDied","Data":"512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455"}
Oct 31 19:02:11.765365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.765333    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht" event={"ID":"8ed1f34a-89a9-4dca-a574-e602d5471ed3","Type":"ContainerDied","Data":"ea67d41eb108193ad5cfc8534b7852c6be21757f5f52f37f6ada695e17e68e84"}
Oct 31 19:02:11.765365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.765362    4484 scope.go:122] "RemoveContainer" containerID="512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455"
Oct 31 19:02:11.766815 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.766793    4484 generic.go:356] "Generic (PLEG): container finished" podID="34bac501-dd5d-499e-80e4-e7016b536662" containerID="c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54" exitCode=0
Oct 31 19:02:11.767097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.766838    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"
Oct 31 19:02:11.767097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.766882    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4" event={"ID":"34bac501-dd5d-499e-80e4-e7016b536662","Type":"ContainerDied","Data":"c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54"}
Oct 31 19:02:11.767097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.766941    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4" event={"ID":"34bac501-dd5d-499e-80e4-e7016b536662","Type":"ContainerDied","Data":"5c8dd38cb55206a78b805c03fbed7188c27688837b5d45dfee6fa73cdf12bcbc"}
Oct 31 19:02:11.768690 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.768666    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq" event={"ID":"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a","Type":"ContainerStarted","Data":"13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6"}
Oct 31 19:02:11.768782 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.768712    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq" event={"ID":"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a","Type":"ContainerStarted","Data":"b7d2de7bb32ca0a089e732a61b6ff63a2293d93b92d1e9e4889676e257661a26"}
Oct 31 19:02:11.776642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.776621    4484 scope.go:122] "RemoveContainer" containerID="512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455"
Oct 31 19:02:11.776975 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:02:11.776951    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455\": container with ID starting with 512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455 not found: ID does not exist" containerID="512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455"
Oct 31 19:02:11.777046 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.776985    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455"} err="failed to get container status \"512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455\": rpc error: code = NotFound desc = could not find container \"512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455\": container with ID starting with 512e6b3485f766c86cbd9e0a97ff2949e574b1aaff3d0ede089eb59e915d5455 not found: ID does not exist"
Oct 31 19:02:11.777046 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.777002    4484 scope.go:122] "RemoveContainer" containerID="c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54"
Oct 31 19:02:11.785813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.785794    4484 scope.go:122] "RemoveContainer" containerID="c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54"
Oct 31 19:02:11.786112 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:02:11.786092    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54\": container with ID starting with c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54 not found: ID does not exist" containerID="c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54"
Oct 31 19:02:11.786176 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.786128    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54"} err="failed to get container status \"c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54\": rpc error: code = NotFound desc = could not find container \"c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54\": container with ID starting with c3969cb9829a0134b2d2952471b36748578a5173e87ab705beb9e5d7c2046d54 not found: ID does not exist"
Oct 31 19:02:11.812383 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.812343    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed1f34a-89a9-4dca-a574-e602d5471ed3-kube-api-access-rn46k\" (UniqueName: \"kubernetes.io/projected/8ed1f34a-89a9-4dca-a574-e602d5471ed3-kube-api-access-rn46k\") pod \"8ed1f34a-89a9-4dca-a574-e602d5471ed3\" (UID: \"8ed1f34a-89a9-4dca-a574-e602d5471ed3\") "
Oct 31 19:02:11.812582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.812407    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed1f34a-89a9-4dca-a574-e602d5471ed3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed1f34a-89a9-4dca-a574-e602d5471ed3-plugins-config-volume\") pod \"8ed1f34a-89a9-4dca-a574-e602d5471ed3\" (UID: \"8ed1f34a-89a9-4dca-a574-e602d5471ed3\") "
Oct 31 19:02:11.812786 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.812765    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed1f34a-89a9-4dca-a574-e602d5471ed3-plugins-config-volume" pod "8ed1f34a-89a9-4dca-a574-e602d5471ed3" (UID: "8ed1f34a-89a9-4dca-a574-e602d5471ed3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:02:11.813005 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.812985    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed1f34a-89a9-4dca-a574-e602d5471ed3-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:11.814825 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.814803    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed1f34a-89a9-4dca-a574-e602d5471ed3-kube-api-access-rn46k" pod "8ed1f34a-89a9-4dca-a574-e602d5471ed3" (UID: "8ed1f34a-89a9-4dca-a574-e602d5471ed3"). InnerVolumeSpecName "kube-api-access-rn46k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:02:11.914184 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.914144    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn46k\" (UniqueName: \"kubernetes.io/projected/8ed1f34a-89a9-4dca-a574-e602d5471ed3-kube-api-access-rn46k\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:02:11.976311 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.976226    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"]
Oct 31 19:02:11.981983 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.981947    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b11e44-5584799bc5-cfkw4"]
Oct 31 19:02:11.993695 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:11.993666    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34bac501-dd5d-499e-80e4-e7016b536662" path="/var/lib/kubelet/pods/34bac501-dd5d-499e-80e4-e7016b536662/volumes"
Oct 31 19:02:12.100423 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:12.100366    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq" podStartSLOduration=2.100350047 podStartE2EDuration="2.100350047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:12.098032451 +0000 UTC m=+2478.244482331" watchObservedRunningTime="2026-10-31 19:02:12.100350047 +0000 UTC m=+2478.246799911"
Oct 31 19:02:12.210654 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:12.210605    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"]
Oct 31 19:02:12.214345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:12.214321    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b11e44-865d5bb667-7r4ht"]
Oct 31 19:02:12.680501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:12.674270    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:12.680501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:12.674443    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 19:02:12.680501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:12.674518    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq" podUID="4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 19:02:12.916320 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:12.916284    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:13.676139 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:13.676096    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 19:02:13.676408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:13.676378    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq" podUID="4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 19:02:13.941341 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:13.941241    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:14.001658 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:14.001623    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed1f34a-89a9-4dca-a574-e602d5471ed3" path="/var/lib/kubelet/pods/8ed1f34a-89a9-4dca-a574-e602d5471ed3/volumes"
Oct 31 19:02:14.235922 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:14.235828    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:14.678501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:14.676922    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 19:02:14.678501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:14.676989    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq" podUID="4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 19:02:15.678502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:15.674924    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 19:02:15.678502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:15.674988    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq" podUID="4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 19:02:16.673974 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:16.673934    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 19:02:16.674326 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:16.674292    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq" podUID="4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 19:02:17.670721 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:17.670685    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 19:02:17.671155 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:17.670764    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq" podUID="4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 19:02:17.956949 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:17.956859    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:02:18.675962 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:18.675926    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:18.676466 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:18.676314    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:18.677198 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:18.677180    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:02:19.777901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:19.777859    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:25.470859 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:02:25.470821    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:02:26.292924 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:26.292859    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:30.867726 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:30.867689    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:35.482263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:35.482231    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:40.276196 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.276152    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:40.354156 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.354124    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"]
Oct 31 19:02:40.354712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.354699    4484 cpu_manager.go:414] "containerMap: removing container" podUID="34bac501-dd5d-499e-80e4-e7016b536662" containerName="payload-processing"
Oct 31 19:02:40.354813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.354714    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34bac501-dd5d-499e-80e4-e7016b536662" containerName="payload-processing"
Oct 31 19:02:40.354813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.354739    4484 cpu_manager.go:414] "containerMap: removing container" podUID="8ed1f34a-89a9-4dca-a574-e602d5471ed3" containerName="payload-pre-processing"
Oct 31 19:02:40.354813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.354744    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed1f34a-89a9-4dca-a574-e602d5471ed3" containerName="payload-pre-processing"
Oct 31 19:02:40.354916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.354840    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="34bac501-dd5d-499e-80e4-e7016b536662" containerName="payload-processing"
Oct 31 19:02:40.354916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.354852    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed1f34a-89a9-4dca-a574-e602d5471ed3" containerName="payload-pre-processing"
Oct 31 19:02:40.358275 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.358256    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.463289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.463251    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcf9q\" (UniqueName: \"kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-kube-api-access-pcf9q\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.463468 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.463389    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.463468 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.463452    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.463648 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.463575    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.463711 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.463643    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.463776 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.463751    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.463815 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.463797    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.463849 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.463832    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.463919 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.463903    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.464524 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.464462    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"]
Oct 31 19:02:40.564692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.564581    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.564692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.564636    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcf9q\" (UniqueName: \"kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-kube-api-access-pcf9q\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.564692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.564689    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.565007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.564716    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.565007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.564748    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.565007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.564795    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.565007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.564839    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.565007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.564865    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.565007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.564898    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.565896 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.565847    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-data\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.566565 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.566416    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.567704 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.567657    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istiod-ca-cert\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.568127 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.568106    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-credential-socket\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.568458 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.568438    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-certs\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.571176 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.571145    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-envoy\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.572002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.571978    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-podinfo\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.622178 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.622142    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-token\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.622365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.622322    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcf9q\" (UniqueName: \"kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-kube-api-access-pcf9q\") pod \"e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") " pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:40.672380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:40.672336    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:41.107660 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:41.107619    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"]
Oct 31 19:02:41.108351 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:02:41.108324    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc8065b_3f0f_4006_b2a2_401cfdb28042.slice/crio-bc83bbd304531f613bc9b83ad8f67ee069c7a0513567c2b1ac706eee74fbbdb5 WatchSource:0}: Error finding container bc83bbd304531f613bc9b83ad8f67ee069c7a0513567c2b1ac706eee74fbbdb5: Status 404 returned error can't find the container with id bc83bbd304531f613bc9b83ad8f67ee069c7a0513567c2b1ac706eee74fbbdb5
Oct 31 19:02:41.110811 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:41.110778    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:02:41.110884 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:41.110851    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:02:41.110921 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:41.110881    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:02:41.977555 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:41.977518    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr" event={"ID":"8bc8065b-3f0f-4006-b2a2-401cfdb28042","Type":"ContainerStarted","Data":"7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125"}
Oct 31 19:02:41.977955 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:41.977605    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr" event={"ID":"8bc8065b-3f0f-4006-b2a2-401cfdb28042","Type":"ContainerStarted","Data":"bc83bbd304531f613bc9b83ad8f67ee069c7a0513567c2b1ac706eee74fbbdb5"}
Oct 31 19:02:42.098091 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:42.098028    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr" podStartSLOduration=2.098014134 podStartE2EDuration="2.098014134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:42.095524279 +0000 UTC m=+2508.241974144" watchObservedRunningTime="2026-10-31 19:02:42.098014134 +0000 UTC m=+2508.244463993"
Oct 31 19:02:42.679500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:42.675899    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:42.684502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:42.680665    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 31 19:02:42.684502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:42.680770    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr" podUID="8bc8065b-3f0f-4006-b2a2-401cfdb28042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 31 19:02:43.675504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:43.674948    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 31 19:02:43.675504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:43.675019    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr" podUID="8bc8065b-3f0f-4006-b2a2-401cfdb28042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 31 19:02:44.678500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:44.674962    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 31 19:02:44.678500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:44.675032    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr" podUID="8bc8065b-3f0f-4006-b2a2-401cfdb28042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 31 19:02:45.107021 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:45.106976    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:45.680121 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:45.680061    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 31 19:02:45.680816 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:45.680775    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr" podUID="8bc8065b-3f0f-4006-b2a2-401cfdb28042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 31 19:02:46.680500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:46.674906    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 31 19:02:46.680500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:46.674973    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr" podUID="8bc8065b-3f0f-4006-b2a2-401cfdb28042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 31 19:02:47.431978 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:47.431941    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:47.673547 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:47.673505    4484 patch_prober.go:29] interesting pod/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 31 19:02:47.673764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:47.673560    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr" podUID="8bc8065b-3f0f-4006-b2a2-401cfdb28042" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 31 19:02:48.677410 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:48.677374    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:48.677838 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:48.677802    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:48.678666 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:48.678632    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:02:48.759929 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:48.759896    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:49.450597 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:49.450560    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:49.623185 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:49.623140    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:51.403619 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:51.403577    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:51.628762 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:51.628718    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:53.360589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:53.360532    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:53.847315 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:53.847278    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:54.401691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:54.401654    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:54.915394 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:54.915362    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:55.469699 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:02:55.469662    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:02:55.797964 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:55.797925    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:57.373446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.373410    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"]
Oct 31 19:02:57.378931 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.378907    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.558191 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.558157    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.558377 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.558215    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.558377 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.558338    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.558453 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.558389    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.558551 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.558527    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.558597 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.558571    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.558597 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.558590    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.558661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.558648    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6477d04c-4bca-4354-8690-fb7e46de5ca2-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.558714 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.558701    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24zzd\" (UniqueName: \"kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-kube-api-access-24zzd\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.574083 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.574047    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"]
Oct 31 19:02:57.660033 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.659936    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.660033 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.659988    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.660033 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.660018    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.660339 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.660040    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.660339 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.660060    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.660339 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.660084    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6477d04c-4bca-4354-8690-fb7e46de5ca2-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.660339 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.660142    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24zzd\" (UniqueName: \"kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-kube-api-access-24zzd\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.660339 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.660191    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.660339 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.660254    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.660692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.660593    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.660692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.660615    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.660848 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.660828    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.660931 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.660905    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.661090 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.661064    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6477d04c-4bca-4354-8690-fb7e46de5ca2-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.662528 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.662497    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.662868 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.662849    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.665122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.665092    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:57.718380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.718349    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.718566 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.718389    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24zzd\" (UniqueName: \"kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-kube-api-access-24zzd\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:57.993079 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:57.993052    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:02:58.421198 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:58.421166    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"]
Oct 31 19:02:58.421718 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:02:58.421447    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6477d04c_4bca_4354_8690_fb7e46de5ca2.slice/crio-efbc2510140b407062f1a4bc73799e94b35d53076ae2f94f46600c52291ec53f WatchSource:0}: Error finding container efbc2510140b407062f1a4bc73799e94b35d53076ae2f94f46600c52291ec53f: Status 404 returned error can't find the container with id efbc2510140b407062f1a4bc73799e94b35d53076ae2f94f46600c52291ec53f
Oct 31 19:02:58.424266 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:58.424231    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:02:58.424367 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:58.424307    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:02:58.424367 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:58.424340    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:02:58.861435 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:58.861401    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:02:59.093720 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:59.093677    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" event={"ID":"6477d04c-4bca-4354-8690-fb7e46de5ca2","Type":"ContainerStarted","Data":"2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4"}
Oct 31 19:02:59.093900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:59.093746    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" event={"ID":"6477d04c-4bca-4354-8690-fb7e46de5ca2","Type":"ContainerStarted","Data":"efbc2510140b407062f1a4bc73799e94b35d53076ae2f94f46600c52291ec53f"}
Oct 31 19:02:59.312947 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:59.312888    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" podStartSLOduration=2.312871368 podStartE2EDuration="2.312871368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:02:59.261255544 +0000 UTC m=+2525.407705410" watchObservedRunningTime="2026-10-31 19:02:59.312871368 +0000 UTC m=+2525.459321233"
Oct 31 19:02:59.998070 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:59.998028    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 31 19:02:59.998571 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:02:59.998126    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" podUID="6477d04c-4bca-4354-8690-fb7e46de5ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 31 19:03:00.005665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:00.005634    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:03:00.993868 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:00.993828    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 31 19:03:00.994079 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:00.993883    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" podUID="6477d04c-4bca-4354-8690-fb7e46de5ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 31 19:03:01.993857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:01.993825    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 31 19:03:01.994231 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:01.993887    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" podUID="6477d04c-4bca-4354-8690-fb7e46de5ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 31 19:03:02.109447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:02.109407    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:02.286027 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:02.285919    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:02.645076 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:02.644986    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:02.993835 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:02.993801    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 31 19:03:02.994015 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:02.993851    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" podUID="6477d04c-4bca-4354-8690-fb7e46de5ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 31 19:03:03.692288 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:03.692249    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:03.993443 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:03.993414    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 31 19:03:03.993666 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:03.993491    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" podUID="6477d04c-4bca-4354-8690-fb7e46de5ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 31 19:03:04.186223 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:04.186122    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:04.993746 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:04.993697    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 31 19:03:04.993946 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:04.993784    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" podUID="6477d04c-4bca-4354-8690-fb7e46de5ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 31 19:03:05.387933 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:05.387841    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:03:06.508070 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:06.508023    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:03:06.509001 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:06.508977    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:03:06.511966 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:06.511938    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:03:07.123645 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.123615    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:07.428929 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.428827    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:07.520843 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.520804    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"]
Oct 31 19:03:07.521397 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.521022    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74" podUID="61eb1d02-c82b-4e2d-8751-a0f318f4e8e2" containerName="payload-pre-processing" containerID="cri-o://5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e" gracePeriod=30
Oct 31 19:03:07.601522 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.587788    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:07.741751 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.741710    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:07.812120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.812091    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"
Oct 31 19:03:07.885696 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.885653    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-plugins-config-volume\") pod \"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2\" (UID: \"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2\") "
Oct 31 19:03:07.885976 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.885745    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-kube-api-access-t76lq\" (UniqueName: \"kubernetes.io/projected/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-kube-api-access-t76lq\") pod \"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2\" (UID: \"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2\") "
Oct 31 19:03:07.886118 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.886097    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-plugins-config-volume" pod "61eb1d02-c82b-4e2d-8751-a0f318f4e8e2" (UID: "61eb1d02-c82b-4e2d-8751-a0f318f4e8e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:03:07.886316 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.886299    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:07.888043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.888010    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-kube-api-access-t76lq" pod "61eb1d02-c82b-4e2d-8751-a0f318f4e8e2" (UID: "61eb1d02-c82b-4e2d-8751-a0f318f4e8e2"). InnerVolumeSpecName "kube-api-access-t76lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:07.987075 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:07.987040    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t76lq\" (UniqueName: \"kubernetes.io/projected/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2-kube-api-access-t76lq\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:08.161167 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:08.161129    4484 generic.go:356] "Generic (PLEG): container finished" podID="61eb1d02-c82b-4e2d-8751-a0f318f4e8e2" containerID="5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e" exitCode=0
Oct 31 19:03:08.161523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:08.161196    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"
Oct 31 19:03:08.161523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:08.161225    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74" event={"ID":"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2","Type":"ContainerDied","Data":"5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e"}
Oct 31 19:03:08.161523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:08.161324    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74" event={"ID":"61eb1d02-c82b-4e2d-8751-a0f318f4e8e2","Type":"ContainerDied","Data":"ee73b978db8656dc0e574033c51d007af39e01ead2cc9e5d83f3cecb1f4c96ad"}
Oct 31 19:03:08.161523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:08.161358    4484 scope.go:122] "RemoveContainer" containerID="5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e"
Oct 31 19:03:08.171997 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:08.171975    4484 scope.go:122] "RemoveContainer" containerID="5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e"
Oct 31 19:03:08.172349 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:03:08.172325    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e\": container with ID starting with 5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e not found: ID does not exist" containerID="5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e"
Oct 31 19:03:08.172420 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:08.172359    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e"} err="failed to get container status \"5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e\": rpc error: code = NotFound desc = could not find container \"5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e\": container with ID starting with 5182f065400378c7fde8d9b93ee5dd6027d7336d6dbe43ce128ea5f85939ec6e not found: ID does not exist"
Oct 31 19:03:08.328069 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:08.327982    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"]
Oct 31 19:03:08.334252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:08.334219    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-713bf0-59f4cdc5cf-dhn74"]
Oct 31 19:03:09.067508 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:09.067448    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:09.217885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:09.217856    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:09.994352 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:09.994315    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61eb1d02-c82b-4e2d-8751-a0f318f4e8e2" path="/var/lib/kubelet/pods/61eb1d02-c82b-4e2d-8751-a0f318f4e8e2/volumes"
Oct 31 19:03:10.530441 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:10.530409    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:10.674980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:10.674945    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:11.029529 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:11.029499    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:11.690913 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:11.690869    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:11.838816 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:11.838772    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:12.289910 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:12.289867    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:12.571737 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:12.571648    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:14.292204 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:14.292161    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:16.107426 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:16.107393    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:16.340780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:16.340745    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:16.516151 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:16.516114    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:17.820875 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:17.820839    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:19.375345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:19.375293    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:19.560303 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:19.560266    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:19.744931 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:19.744896    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:19.956314 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:19.956275    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"]
Oct 31 19:03:19.956845 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:19.956831    4484 cpu_manager.go:414] "containerMap: removing container" podUID="61eb1d02-c82b-4e2d-8751-a0f318f4e8e2" containerName="payload-pre-processing"
Oct 31 19:03:19.956891 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:19.956847    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61eb1d02-c82b-4e2d-8751-a0f318f4e8e2" containerName="payload-pre-processing"
Oct 31 19:03:19.956966 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:19.956957    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="61eb1d02-c82b-4e2d-8751-a0f318f4e8e2" containerName="payload-pre-processing"
Oct 31 19:03:19.961801 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:19.961780    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"
Oct 31 19:03:20.016592 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.016510    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"]
Oct 31 19:03:20.051642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.051517    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6t8n\" (UniqueName: \"kubernetes.io/projected/9fe44e72-b397-4171-9952-56f02961c069-kube-api-access-b6t8n\") pod \"payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b\" (UID: \"9fe44e72-b397-4171-9952-56f02961c069\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"
Oct 31 19:03:20.052120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.052014    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe44e72-b397-4171-9952-56f02961c069-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b\" (UID: \"9fe44e72-b397-4171-9952-56f02961c069\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"
Oct 31 19:03:20.067504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.064855    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-1ef01f-dockercfg-8f2jt\""
Oct 31 19:03:20.067504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.064955    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-1ef01f\""
Oct 31 19:03:20.118581 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.118553    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"]
Oct 31 19:03:20.122505 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.122465    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"
Oct 31 19:03:20.152950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.152915    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe44e72-b397-4171-9952-56f02961c069-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b\" (UID: \"9fe44e72-b397-4171-9952-56f02961c069\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"
Oct 31 19:03:20.153145 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.153017    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6t8n\" (UniqueName: \"kubernetes.io/projected/9fe44e72-b397-4171-9952-56f02961c069-kube-api-access-b6t8n\") pod \"payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b\" (UID: \"9fe44e72-b397-4171-9952-56f02961c069\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"
Oct 31 19:03:20.153667 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.153647    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe44e72-b397-4171-9952-56f02961c069-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b\" (UID: \"9fe44e72-b397-4171-9952-56f02961c069\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"
Oct 31 19:03:20.164640 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.164607    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"]
Oct 31 19:03:20.210770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.210740    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6t8n\" (UniqueName: \"kubernetes.io/projected/9fe44e72-b397-4171-9952-56f02961c069-kube-api-access-b6t8n\") pod \"payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b\" (UID: \"9fe44e72-b397-4171-9952-56f02961c069\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"
Oct 31 19:03:20.254049 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.254015    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6x94\" (UniqueName: \"kubernetes.io/projected/8056bf6d-b304-4656-af4c-8a5b8708845b-kube-api-access-q6x94\") pod \"payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9\" (UID: \"8056bf6d-b304-4656-af4c-8a5b8708845b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"
Oct 31 19:03:20.254225 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.254169    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8056bf6d-b304-4656-af4c-8a5b8708845b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9\" (UID: \"8056bf6d-b304-4656-af4c-8a5b8708845b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"
Oct 31 19:03:20.276028 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.275945    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"
Oct 31 19:03:20.355417 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.355368    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8056bf6d-b304-4656-af4c-8a5b8708845b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9\" (UID: \"8056bf6d-b304-4656-af4c-8a5b8708845b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"
Oct 31 19:03:20.355628 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.355577    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6x94\" (UniqueName: \"kubernetes.io/projected/8056bf6d-b304-4656-af4c-8a5b8708845b-kube-api-access-q6x94\") pod \"payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9\" (UID: \"8056bf6d-b304-4656-af4c-8a5b8708845b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"
Oct 31 19:03:20.356283 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.356257    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8056bf6d-b304-4656-af4c-8a5b8708845b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9\" (UID: \"8056bf6d-b304-4656-af4c-8a5b8708845b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"
Oct 31 19:03:20.413918 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.413890    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6x94\" (UniqueName: \"kubernetes.io/projected/8056bf6d-b304-4656-af4c-8a5b8708845b-kube-api-access-q6x94\") pod \"payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9\" (UID: \"8056bf6d-b304-4656-af4c-8a5b8708845b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"
Oct 31 19:03:20.436911 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.436879    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"
Oct 31 19:03:20.652587 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.652556    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"]
Oct 31 19:03:20.653468 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:03:20.653439    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe44e72_b397_4171_9952_56f02961c069.slice/crio-1471322de4014f664dd1e87364f450e83577ed05a3c56b70554e23e1b63fac7d WatchSource:0}: Error finding container 1471322de4014f664dd1e87364f450e83577ed05a3c56b70554e23e1b63fac7d: Status 404 returned error can't find the container with id 1471322de4014f664dd1e87364f450e83577ed05a3c56b70554e23e1b63fac7d
Oct 31 19:03:20.816279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:20.816250    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"]
Oct 31 19:03:20.816963 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:03:20.816937    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8056bf6d_b304_4656_af4c_8a5b8708845b.slice/crio-c514882708f5468071130bd04cf52ff2d8faa3e5f478da29895707117a157b19 WatchSource:0}: Error finding container c514882708f5468071130bd04cf52ff2d8faa3e5f478da29895707117a157b19: Status 404 returned error can't find the container with id c514882708f5468071130bd04cf52ff2d8faa3e5f478da29895707117a157b19
Oct 31 19:03:21.242728 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:21.242687    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b" event={"ID":"9fe44e72-b397-4171-9952-56f02961c069","Type":"ContainerStarted","Data":"c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e"}
Oct 31 19:03:21.242945 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:21.242754    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b" event={"ID":"9fe44e72-b397-4171-9952-56f02961c069","Type":"ContainerStarted","Data":"1471322de4014f664dd1e87364f450e83577ed05a3c56b70554e23e1b63fac7d"}
Oct 31 19:03:21.242945 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:21.242784    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"
Oct 31 19:03:21.244542 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:21.244512    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9" event={"ID":"8056bf6d-b304-4656-af4c-8a5b8708845b","Type":"ContainerStarted","Data":"c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680"}
Oct 31 19:03:21.244736 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:21.244564    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9" event={"ID":"8056bf6d-b304-4656-af4c-8a5b8708845b","Type":"ContainerStarted","Data":"c514882708f5468071130bd04cf52ff2d8faa3e5f478da29895707117a157b19"}
Oct 31 19:03:21.244736 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:21.244649    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"
Oct 31 19:03:21.359081 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:21.359022    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b" podStartSLOduration=2.359009426 podStartE2EDuration="2.359009426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:21.358063285 +0000 UTC m=+2547.504513149" watchObservedRunningTime="2026-10-31 19:03:21.359009426 +0000 UTC m=+2547.505459308"
Oct 31 19:03:21.473834 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:21.473781    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9" podStartSLOduration=1.47376745 podStartE2EDuration="1.47376745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:21.472704097 +0000 UTC m=+2547.619153962" watchObservedRunningTime="2026-10-31 19:03:21.47376745 +0000 UTC m=+2547.620217340"
Oct 31 19:03:23.840096 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:23.840063    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:25.471289 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:03:25.471253    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:03:25.667880 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:25.667845    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:29.579458 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:29.579420    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:03:30.118357 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.118320    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"]
Oct 31 19:03:30.122553 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.122535    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:30.172979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.172936    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f69e81d-vg59l\" (UID: \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:30.172979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.172976    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f69e81d-vg59l\" (UID: \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:30.173245 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.172999    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcdjv\" (UniqueName: \"kubernetes.io/projected/74c78898-1640-4522-8ebd-5d8fe9a0a49a-kube-api-access-wcdjv\") pod \"maas-api-revoke-sub-2f69e81d-vg59l\" (UID: \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:30.178520 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.178488    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"]
Oct 31 19:03:30.225341 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.225310    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:03:30.225586 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.225543    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:03:30.274694 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.274661    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcdjv\" (UniqueName: \"kubernetes.io/projected/74c78898-1640-4522-8ebd-5d8fe9a0a49a-kube-api-access-wcdjv\") pod \"maas-api-revoke-sub-2f69e81d-vg59l\" (UID: \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:30.274874 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.274831    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f69e81d-vg59l\" (UID: \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:30.274874 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.274858    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f69e81d-vg59l\" (UID: \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:30.275657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.275638    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f69e81d-vg59l\" (UID: \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:30.277440 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.277412    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f69e81d-vg59l\" (UID: \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:30.332255 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.332229    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcdjv\" (UniqueName: \"kubernetes.io/projected/74c78898-1640-4522-8ebd-5d8fe9a0a49a-kube-api-access-wcdjv\") pod \"maas-api-revoke-sub-2f69e81d-vg59l\" (UID: \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:30.437586 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.437466    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:30.455438 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.455390    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:30.833667 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:30.833634    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"]
Oct 31 19:03:31.351013 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:31.350886    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l" event={"ID":"74c78898-1640-4522-8ebd-5d8fe9a0a49a","Type":"ContainerStarted","Data":"87bec9743e06cf4b668a50e5085ff183d876b6fd788799696db18e6f388d2b3f"}
Oct 31 19:03:32.203760 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:32.203720    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:32.253485 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:32.253433    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"
Oct 31 19:03:32.253724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:32.253663    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"
Oct 31 19:03:32.359503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:32.359443    4484 generic.go:356] "Generic (PLEG): container finished" podID="74c78898-1640-4522-8ebd-5d8fe9a0a49a" containerID="69d3a4f95b0e53c89285e45b3186a4c9406f174432edb3936c8ad4095a64dbf3" exitCode=0
Oct 31 19:03:32.359675 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:32.359506    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l" event={"ID":"74c78898-1640-4522-8ebd-5d8fe9a0a49a","Type":"ContainerDied","Data":"69d3a4f95b0e53c89285e45b3186a4c9406f174432edb3936c8ad4095a64dbf3"}
Oct 31 19:03:32.410131 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:32.410093    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:33.520774 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:33.520744    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:33.571059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:33.571024    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-openshift-ca\") pod \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\" (UID: \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\") "
Oct 31 19:03:33.571266 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:33.571072    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-xks-ca\") pod \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\" (UID: \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\") "
Oct 31 19:03:33.571266 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:33.571100    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c78898-1640-4522-8ebd-5d8fe9a0a49a-kube-api-access-wcdjv\" (UniqueName: \"kubernetes.io/projected/74c78898-1640-4522-8ebd-5d8fe9a0a49a-kube-api-access-wcdjv\") pod \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\" (UID: \"74c78898-1640-4522-8ebd-5d8fe9a0a49a\") "
Oct 31 19:03:33.571508 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:33.571449    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-openshift-ca" pod "74c78898-1640-4522-8ebd-5d8fe9a0a49a" (UID: "74c78898-1640-4522-8ebd-5d8fe9a0a49a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:03:33.573517 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:33.573492    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-xks-ca" pod "74c78898-1640-4522-8ebd-5d8fe9a0a49a" (UID: "74c78898-1640-4522-8ebd-5d8fe9a0a49a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:03:33.573761 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:33.573730    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c78898-1640-4522-8ebd-5d8fe9a0a49a-kube-api-access-wcdjv" pod "74c78898-1640-4522-8ebd-5d8fe9a0a49a" (UID: "74c78898-1640-4522-8ebd-5d8fe9a0a49a"). InnerVolumeSpecName "kube-api-access-wcdjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:33.672482 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:33.672380    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:33.672482 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:33.672412    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c78898-1640-4522-8ebd-5d8fe9a0a49a-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:33.672482 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:33.672425    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcdjv\" (UniqueName: \"kubernetes.io/projected/74c78898-1640-4522-8ebd-5d8fe9a0a49a-kube-api-access-wcdjv\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:33.826446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:33.826406    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:34.301519 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:34.301407    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"]
Oct 31 19:03:34.301704 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:34.301667    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq" podUID="4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" containerName="istio-proxy" containerID="cri-o://13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6" gracePeriod=30
Oct 31 19:03:34.307536 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:34.307497    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:34.309465 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:34.309443    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"]
Oct 31 19:03:34.309683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:34.309664    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr" podUID="8bc8065b-3f0f-4006-b2a2-401cfdb28042" containerName="istio-proxy" containerID="cri-o://7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125" gracePeriod=30
Oct 31 19:03:34.376697 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:34.376537    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l" event={"ID":"74c78898-1640-4522-8ebd-5d8fe9a0a49a","Type":"ContainerDied","Data":"87bec9743e06cf4b668a50e5085ff183d876b6fd788799696db18e6f388d2b3f"}
Oct 31 19:03:34.376697 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:34.376634    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87bec9743e06cf4b668a50e5085ff183d876b6fd788799696db18e6f388d2b3f"
Oct 31 19:03:34.377018 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:34.376719    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"
Oct 31 19:03:35.495249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:35.495207    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:35.892737 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:35.892622    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:36.474750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:36.474713    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:36.673549 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:36.673513    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:39.571736 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.571670    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:03:39.625567 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.625535    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:03:39.652053 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652022    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-envoy\") pod \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") "
Oct 31 19:03:39.652256 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652104    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istiod-ca-cert\") pod \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") "
Oct 31 19:03:39.652256 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652124    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-token\" (UniqueName: \"kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-token\") pod \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") "
Oct 31 19:03:39.652256 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652163    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-certs\") pod \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") "
Oct 31 19:03:39.652256 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652189    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-credential-socket\") pod \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") "
Oct 31 19:03:39.652256 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652216    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-podinfo\") pod \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") "
Oct 31 19:03:39.652256 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652239    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-kube-api-access-b6fk8\" (UniqueName: \"kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-kube-api-access-b6fk8\") pod \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") "
Oct 31 19:03:39.652602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652315    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-socket\") pod \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") "
Oct 31 19:03:39.652602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652376    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-data\") pod \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\" (UID: \"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a\") "
Oct 31 19:03:39.652602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652427    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-certs" pod "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" (UID: "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.652767 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652691    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-socket" pod "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" (UID: "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.652767 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652702    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istiod-ca-cert" pod "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" (UID: "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:03:39.652941 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652885    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-credential-socket" pod "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" (UID: "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.652941 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652901    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-data" pod "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" (UID: "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.653155 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.652994    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.653155 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.653011    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.653155 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.653026    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.653155 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.653040    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.654606 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.654582    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-envoy" pod "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" (UID: "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.654733 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.654630    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-token" pod "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" (UID: "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:39.655226 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.655207    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-kube-api-access-b6fk8" pod "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" (UID: "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a"). InnerVolumeSpecName "kube-api-access-b6fk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:39.655226 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.655208    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-podinfo" pod "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" (UID: "4be5dd8d-6d5a-4ed4-a5aa-23094889d17a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:03:39.754409 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.754377    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-data\") pod \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") "
Oct 31 19:03:39.754616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.754453    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-credential-socket\") pod \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") "
Oct 31 19:03:39.754616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.754518    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-envoy\") pod \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") "
Oct 31 19:03:39.754616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.754547    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-certs\") pod \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") "
Oct 31 19:03:39.754616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.754611    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istiod-ca-cert\") pod \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") "
Oct 31 19:03:39.754821 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.754635    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-podinfo\") pod \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") "
Oct 31 19:03:39.754821 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.754699    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-socket\") pod \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") "
Oct 31 19:03:39.754821 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.754789    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-kube-api-access-pcf9q\" (UniqueName: \"kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-kube-api-access-pcf9q\") pod \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") "
Oct 31 19:03:39.754981 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.754783    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-data" pod "8bc8065b-3f0f-4006-b2a2-401cfdb28042" (UID: "8bc8065b-3f0f-4006-b2a2-401cfdb28042"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.754981 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.754843    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-token\" (UniqueName: \"kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-token\") pod \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\" (UID: \"8bc8065b-3f0f-4006-b2a2-401cfdb28042\") "
Oct 31 19:03:39.754981 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.754856    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-credential-socket" pod "8bc8065b-3f0f-4006-b2a2-401cfdb28042" (UID: "8bc8065b-3f0f-4006-b2a2-401cfdb28042"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.755152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.755024    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-socket" pod "8bc8065b-3f0f-4006-b2a2-401cfdb28042" (UID: "8bc8065b-3f0f-4006-b2a2-401cfdb28042"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.755269 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.755245    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-certs" pod "8bc8065b-3f0f-4006-b2a2-401cfdb28042" (UID: "8bc8065b-3f0f-4006-b2a2-401cfdb28042"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.755447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.755351    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.755447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.755370    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.755447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.755384    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6fk8\" (UniqueName: \"kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-kube-api-access-b6fk8\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.755447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.755399    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.755447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.755412    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.755447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.755425    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.755447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.755439    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.755987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.755451    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.755987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.755464    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.755987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.755886    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istiod-ca-cert" pod "8bc8065b-3f0f-4006-b2a2-401cfdb28042" (UID: "8bc8065b-3f0f-4006-b2a2-401cfdb28042"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:03:39.757678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.757558    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-envoy" pod "8bc8065b-3f0f-4006-b2a2-401cfdb28042" (UID: "8bc8065b-3f0f-4006-b2a2-401cfdb28042"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:03:39.757678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.757617    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-kube-api-access-pcf9q" pod "8bc8065b-3f0f-4006-b2a2-401cfdb28042" (UID: "8bc8065b-3f0f-4006-b2a2-401cfdb28042"). InnerVolumeSpecName "kube-api-access-pcf9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:39.757678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.757653    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-token" pod "8bc8065b-3f0f-4006-b2a2-401cfdb28042" (UID: "8bc8065b-3f0f-4006-b2a2-401cfdb28042"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:39.757915 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.757899    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-podinfo" pod "8bc8065b-3f0f-4006-b2a2-401cfdb28042" (UID: "8bc8065b-3f0f-4006-b2a2-401cfdb28042"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:03:39.856682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.856571    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.856682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.856605    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.856682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.856623    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.856682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.856637    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcf9q\" (UniqueName: \"kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-kube-api-access-pcf9q\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:39.856682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:39.856652    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bc8065b-3f0f-4006-b2a2-401cfdb28042-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:40.285372 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.285329    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:40.417238 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.417202    4484 generic.go:356] "Generic (PLEG): container finished" podID="8bc8065b-3f0f-4006-b2a2-401cfdb28042" containerID="7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125" exitCode=0
Oct 31 19:03:40.417449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.417289    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"
Oct 31 19:03:40.417449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.417293    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr" event={"ID":"8bc8065b-3f0f-4006-b2a2-401cfdb28042","Type":"ContainerDied","Data":"7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125"}
Oct 31 19:03:40.417449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.417353    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr" event={"ID":"8bc8065b-3f0f-4006-b2a2-401cfdb28042","Type":"ContainerDied","Data":"bc83bbd304531f613bc9b83ad8f67ee069c7a0513567c2b1ac706eee74fbbdb5"}
Oct 31 19:03:40.417449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.417380    4484 scope.go:122] "RemoveContainer" containerID="7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125"
Oct 31 19:03:40.419022 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.419000    4484 generic.go:356] "Generic (PLEG): container finished" podID="4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" containerID="13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6" exitCode=0
Oct 31 19:03:40.419193 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.419107    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq" event={"ID":"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a","Type":"ContainerDied","Data":"13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6"}
Oct 31 19:03:40.419329 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.419222    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq" event={"ID":"4be5dd8d-6d5a-4ed4-a5aa-23094889d17a","Type":"ContainerDied","Data":"b7d2de7bb32ca0a089e732a61b6ff63a2293d93b92d1e9e4889676e257661a26"}
Oct 31 19:03:40.419329 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.419256    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"
Oct 31 19:03:40.427827 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.427802    4484 scope.go:122] "RemoveContainer" containerID="7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125"
Oct 31 19:03:40.428151 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:03:40.428131    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125\": container with ID starting with 7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125 not found: ID does not exist" containerID="7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125"
Oct 31 19:03:40.428233 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.428165    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125"} err="failed to get container status \"7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125\": rpc error: code = NotFound desc = could not find container \"7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125\": container with ID starting with 7ecacc851e47aff791940cfd94b3950789749ca889f43f8b3594daf5fc0b4125 not found: ID does not exist"
Oct 31 19:03:40.428233 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.428193    4484 scope.go:122] "RemoveContainer" containerID="13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6"
Oct 31 19:03:40.437661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.437520    4484 scope.go:122] "RemoveContainer" containerID="13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6"
Oct 31 19:03:40.437911 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:03:40.437886    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6\": container with ID starting with 13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6 not found: ID does not exist" containerID="13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6"
Oct 31 19:03:40.437993 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.437921    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6"} err="failed to get container status \"13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6\": rpc error: code = NotFound desc = could not find container \"13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6\": container with ID starting with 13463f6b5dc10ac05c537b92a4c3ee8fb16137654eb3bddc22e7cc737a7ae9f6 not found: ID does not exist"
Oct 31 19:03:40.463174 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.463140    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:40.582895 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.582804    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"]
Oct 31 19:03:40.624236 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.624199    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:40.633288 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.633241    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-x7tqr"]
Oct 31 19:03:40.693292 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.693251    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"]
Oct 31 19:03:40.695701 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.695680    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4c58c5-openshift-default-5b9f75bf65-l57cq"]
Oct 31 19:03:40.786287 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:40.786248    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:41.994815 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:41.994783    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" path="/var/lib/kubelet/pods/4be5dd8d-6d5a-4ed4-a5aa-23094889d17a/volumes"
Oct 31 19:03:41.995247 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:41.995234    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc8065b-3f0f-4006-b2a2-401cfdb28042" path="/var/lib/kubelet/pods/8bc8065b-3f0f-4006-b2a2-401cfdb28042/volumes"
Oct 31 19:03:45.125910 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.125875    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:45.141232 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.141202    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"]
Oct 31 19:03:45.141991 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.141975    4484 cpu_manager.go:414] "containerMap: removing container" podUID="4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" containerName="istio-proxy"
Oct 31 19:03:45.142048 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.141994    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" containerName="istio-proxy"
Oct 31 19:03:45.142048 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.142035    4484 cpu_manager.go:414] "containerMap: removing container" podUID="8bc8065b-3f0f-4006-b2a2-401cfdb28042" containerName="istio-proxy"
Oct 31 19:03:45.142048 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.142043    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc8065b-3f0f-4006-b2a2-401cfdb28042" containerName="istio-proxy"
Oct 31 19:03:45.142150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.142081    4484 cpu_manager.go:414] "containerMap: removing container" podUID="74c78898-1640-4522-8ebd-5d8fe9a0a49a" containerName="revoke-keys"
Oct 31 19:03:45.142150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.142089    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c78898-1640-4522-8ebd-5d8fe9a0a49a" containerName="revoke-keys"
Oct 31 19:03:45.142214 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.142197    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc8065b-3f0f-4006-b2a2-401cfdb28042" containerName="istio-proxy"
Oct 31 19:03:45.142250 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.142216    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c78898-1640-4522-8ebd-5d8fe9a0a49a" containerName="revoke-keys"
Oct 31 19:03:45.142250 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.142228    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be5dd8d-6d5a-4ed4-a5aa-23094889d17a" containerName="istio-proxy"
Oct 31 19:03:45.147416 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.147395    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:45.199936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.199902    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"]
Oct 31 19:03:45.218993 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.218923    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wr9d\" (UniqueName: \"kubernetes.io/projected/359d0aca-1468-4717-8363-508cf99704b4-kube-api-access-5wr9d\") pod \"maas-api-revoke-sub-19bfd618-tk46n\" (UID: \"359d0aca-1468-4717-8363-508cf99704b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:45.219234 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.219059    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359d0aca-1468-4717-8363-508cf99704b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19bfd618-tk46n\" (UID: \"359d0aca-1468-4717-8363-508cf99704b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:45.219234 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.219101    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359d0aca-1468-4717-8363-508cf99704b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19bfd618-tk46n\" (UID: \"359d0aca-1468-4717-8363-508cf99704b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:45.247834 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.247787    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:03:45.248032 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.247944    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:03:45.319978 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.319945    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wr9d\" (UniqueName: \"kubernetes.io/projected/359d0aca-1468-4717-8363-508cf99704b4-kube-api-access-5wr9d\") pod \"maas-api-revoke-sub-19bfd618-tk46n\" (UID: \"359d0aca-1468-4717-8363-508cf99704b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:45.320136 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.320023    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359d0aca-1468-4717-8363-508cf99704b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19bfd618-tk46n\" (UID: \"359d0aca-1468-4717-8363-508cf99704b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:45.320136 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.320057    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359d0aca-1468-4717-8363-508cf99704b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19bfd618-tk46n\" (UID: \"359d0aca-1468-4717-8363-508cf99704b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:45.320787 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.320763    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359d0aca-1468-4717-8363-508cf99704b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19bfd618-tk46n\" (UID: \"359d0aca-1468-4717-8363-508cf99704b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:45.322401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.322376    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359d0aca-1468-4717-8363-508cf99704b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19bfd618-tk46n\" (UID: \"359d0aca-1468-4717-8363-508cf99704b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:45.324963 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.324935    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:45.377426 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.377354    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wr9d\" (UniqueName: \"kubernetes.io/projected/359d0aca-1468-4717-8363-508cf99704b4-kube-api-access-5wr9d\") pod \"maas-api-revoke-sub-19bfd618-tk46n\" (UID: \"359d0aca-1468-4717-8363-508cf99704b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:45.460884 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.460856    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:45.660163 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.660078    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"]
Oct 31 19:03:45.664260 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.664243    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"
Oct 31 19:03:45.722109 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.718266    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a2c587\""
Oct 31 19:03:45.722529 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.722503    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"]
Oct 31 19:03:45.764699 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.764671    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a2c587-dockercfg-52fkj\""
Oct 31 19:03:45.801206 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.801165    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:45.837206 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.837176    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvk7c\" (UniqueName: \"kubernetes.io/projected/62fe1994-2aa0-434b-b225-2d61b887b9ee-kube-api-access-xvk7c\") pod \"payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr\" (UID: \"62fe1994-2aa0-434b-b225-2d61b887b9ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"
Oct 31 19:03:45.837371 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.837339    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62fe1994-2aa0-434b-b225-2d61b887b9ee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr\" (UID: \"62fe1994-2aa0-434b-b225-2d61b887b9ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"
Oct 31 19:03:45.837749 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.837732    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"]
Oct 31 19:03:45.838354 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:03:45.838322    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359d0aca_1468_4717_8363_508cf99704b4.slice/crio-a5b8073e42d71ea80b232edb5e8b87e42848d2dbcb2488f97e3f7d3b6df0970f WatchSource:0}: Error finding container a5b8073e42d71ea80b232edb5e8b87e42848d2dbcb2488f97e3f7d3b6df0970f: Status 404 returned error can't find the container with id a5b8073e42d71ea80b232edb5e8b87e42848d2dbcb2488f97e3f7d3b6df0970f
Oct 31 19:03:45.938884 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.938790    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62fe1994-2aa0-434b-b225-2d61b887b9ee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr\" (UID: \"62fe1994-2aa0-434b-b225-2d61b887b9ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"
Oct 31 19:03:45.939041 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.938894    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvk7c\" (UniqueName: \"kubernetes.io/projected/62fe1994-2aa0-434b-b225-2d61b887b9ee-kube-api-access-xvk7c\") pod \"payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr\" (UID: \"62fe1994-2aa0-434b-b225-2d61b887b9ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"
Oct 31 19:03:45.939615 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.939597    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62fe1994-2aa0-434b-b225-2d61b887b9ee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr\" (UID: \"62fe1994-2aa0-434b-b225-2d61b887b9ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"
Oct 31 19:03:45.995740 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:45.995712    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvk7c\" (UniqueName: \"kubernetes.io/projected/62fe1994-2aa0-434b-b225-2d61b887b9ee-kube-api-access-xvk7c\") pod \"payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr\" (UID: \"62fe1994-2aa0-434b-b225-2d61b887b9ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"
Oct 31 19:03:46.279052 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:46.279014    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"
Oct 31 19:03:46.469881 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:46.469846    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n" event={"ID":"359d0aca-1468-4717-8363-508cf99704b4","Type":"ContainerStarted","Data":"eeb953e872ea1ce90d71557a26814e4d0321cb4e34bf65be5e3e803e1dee1fc3"}
Oct 31 19:03:46.470069 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:46.469902    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n" event={"ID":"359d0aca-1468-4717-8363-508cf99704b4","Type":"ContainerStarted","Data":"a5b8073e42d71ea80b232edb5e8b87e42848d2dbcb2488f97e3f7d3b6df0970f"}
Oct 31 19:03:46.585642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:46.585533    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n" podStartSLOduration=1.585516121 podStartE2EDuration="1.585516121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:46.583708685 +0000 UTC m=+2572.730158551" watchObservedRunningTime="2026-10-31 19:03:46.585516121 +0000 UTC m=+2572.731965985"
Oct 31 19:03:46.696875 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:46.696847    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"]
Oct 31 19:03:46.697741 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:03:46.697711    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fe1994_2aa0_434b_b225_2d61b887b9ee.slice/crio-fb8f9bb86528ccf1675128729e5c0bafcc954b147f0b14fba824f4a95af6d9be WatchSource:0}: Error finding container fb8f9bb86528ccf1675128729e5c0bafcc954b147f0b14fba824f4a95af6d9be: Status 404 returned error can't find the container with id fb8f9bb86528ccf1675128729e5c0bafcc954b147f0b14fba824f4a95af6d9be
Oct 31 19:03:47.477608 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:47.477568    4484 generic.go:356] "Generic (PLEG): container finished" podID="359d0aca-1468-4717-8363-508cf99704b4" containerID="eeb953e872ea1ce90d71557a26814e4d0321cb4e34bf65be5e3e803e1dee1fc3" exitCode=0
Oct 31 19:03:47.478087 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:47.477649    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n" event={"ID":"359d0aca-1468-4717-8363-508cf99704b4","Type":"ContainerDied","Data":"eeb953e872ea1ce90d71557a26814e4d0321cb4e34bf65be5e3e803e1dee1fc3"}
Oct 31 19:03:47.479446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:47.479413    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr" event={"ID":"62fe1994-2aa0-434b-b225-2d61b887b9ee","Type":"ContainerStarted","Data":"eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030"}
Oct 31 19:03:47.479593 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:47.479462    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr" event={"ID":"62fe1994-2aa0-434b-b225-2d61b887b9ee","Type":"ContainerStarted","Data":"fb8f9bb86528ccf1675128729e5c0bafcc954b147f0b14fba824f4a95af6d9be"}
Oct 31 19:03:47.479593 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:47.479538    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"
Oct 31 19:03:47.713447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:47.713393    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr" podStartSLOduration=2.713378285 podStartE2EDuration="2.713378285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:47.710678486 +0000 UTC m=+2573.857128351" watchObservedRunningTime="2026-10-31 19:03:47.713378285 +0000 UTC m=+2573.859828210"
Oct 31 19:03:48.642129 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:48.642098    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:48.790498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:48.789525    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359d0aca-1468-4717-8363-508cf99704b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359d0aca-1468-4717-8363-508cf99704b4-maas-api-openshift-ca\") pod \"359d0aca-1468-4717-8363-508cf99704b4\" (UID: \"359d0aca-1468-4717-8363-508cf99704b4\") "
Oct 31 19:03:48.790498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:48.789759    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359d0aca-1468-4717-8363-508cf99704b4-kube-api-access-5wr9d\" (UniqueName: \"kubernetes.io/projected/359d0aca-1468-4717-8363-508cf99704b4-kube-api-access-5wr9d\") pod \"359d0aca-1468-4717-8363-508cf99704b4\" (UID: \"359d0aca-1468-4717-8363-508cf99704b4\") "
Oct 31 19:03:48.790498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:48.789796    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/359d0aca-1468-4717-8363-508cf99704b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359d0aca-1468-4717-8363-508cf99704b4-maas-api-xks-ca\") pod \"359d0aca-1468-4717-8363-508cf99704b4\" (UID: \"359d0aca-1468-4717-8363-508cf99704b4\") "
Oct 31 19:03:48.790498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:48.789958    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359d0aca-1468-4717-8363-508cf99704b4-maas-api-openshift-ca" pod "359d0aca-1468-4717-8363-508cf99704b4" (UID: "359d0aca-1468-4717-8363-508cf99704b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:03:48.790498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:48.790166    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359d0aca-1468-4717-8363-508cf99704b4-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:48.792864 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:48.792836    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359d0aca-1468-4717-8363-508cf99704b4-kube-api-access-5wr9d" pod "359d0aca-1468-4717-8363-508cf99704b4" (UID: "359d0aca-1468-4717-8363-508cf99704b4"). InnerVolumeSpecName "kube-api-access-5wr9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:03:48.802650 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:48.802610    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/359d0aca-1468-4717-8363-508cf99704b4-maas-api-xks-ca" pod "359d0aca-1468-4717-8363-508cf99704b4" (UID: "359d0aca-1468-4717-8363-508cf99704b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:03:48.890941 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:48.890888    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wr9d\" (UniqueName: \"kubernetes.io/projected/359d0aca-1468-4717-8363-508cf99704b4-kube-api-access-5wr9d\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:48.890941 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:48.890918    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359d0aca-1468-4717-8363-508cf99704b4-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:03:49.496286 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:49.496244    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n" event={"ID":"359d0aca-1468-4717-8363-508cf99704b4","Type":"ContainerDied","Data":"a5b8073e42d71ea80b232edb5e8b87e42848d2dbcb2488f97e3f7d3b6df0970f"}
Oct 31 19:03:49.496286 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:49.496280    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"
Oct 31 19:03:49.496514 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:49.496311    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5b8073e42d71ea80b232edb5e8b87e42848d2dbcb2488f97e3f7d3b6df0970f"
Oct 31 19:03:50.620731 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:50.620693    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:50.776156 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:50.776122    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:50.927122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:50.927030    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:54.737431 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:54.737385    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:55.423687 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:55.423591    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:55.475535 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:03:55.475498    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:03:57.823150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.823116    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"]
Oct 31 19:03:57.823691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.823676    4484 cpu_manager.go:414] "containerMap: removing container" podUID="359d0aca-1468-4717-8363-508cf99704b4" containerName="revoke-keys"
Oct 31 19:03:57.823755 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.823693    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359d0aca-1468-4717-8363-508cf99704b4" containerName="revoke-keys"
Oct 31 19:03:57.823812 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.823804    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="359d0aca-1468-4717-8363-508cf99704b4" containerName="revoke-keys"
Oct 31 19:03:57.828536 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.828520    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.892194 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.892147    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.892194 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.892190    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.892424 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.892280    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.892424 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.892334    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.892424 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.892401    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.892712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.892429    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4422bb-55c9-4ba7-b963-850377365978-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.892712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.892549    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4422bb-55c9-4ba7-b963-850377365978-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.892712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.892579    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.892712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.892666    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2qgg\" (UniqueName: \"kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-kube-api-access-v2qgg\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.993968 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.993934    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.993968 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.993970    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.994249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.994027    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.994249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.994053    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4422bb-55c9-4ba7-b963-850377365978-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.994249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.994108    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4422bb-55c9-4ba7-b963-850377365978-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.994249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.994133    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.994249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.994188    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2qgg\" (UniqueName: \"kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-kube-api-access-v2qgg\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.994618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.994319    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.994618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.994349    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.994618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.994415    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-certs\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.994784 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.994731    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-credential-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.995076 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.995011    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-data\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.995076 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.995032    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4422bb-55c9-4ba7-b963-850377365978-istiod-ca-cert\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.995463 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.995441    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-socket\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.997000 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.996977    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4422bb-55c9-4ba7-b963-850377365978-istio-podinfo\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:57.997672 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:57.997652    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-envoy\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:58.035913 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:58.035877    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"]
Oct 31 19:03:58.052315 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:58.052275    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-istio-token\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:58.052525 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:58.052439    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2qgg\" (UniqueName: \"kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-kube-api-access-v2qgg\") pod \"e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") " pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:58.142887 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:58.142794    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:03:58.489293 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:58.489255    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"
Oct 31 19:03:58.574445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:58.574417    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"]
Oct 31 19:03:58.574794 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:03:58.574766    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4422bb_55c9_4ba7_b963_850377365978.slice/crio-37756a208de787537b17435f7026924a9a7e39cc1f481162f517d31666fa3f25 WatchSource:0}: Error finding container 37756a208de787537b17435f7026924a9a7e39cc1f481162f517d31666fa3f25: Status 404 returned error can't find the container with id 37756a208de787537b17435f7026924a9a7e39cc1f481162f517d31666fa3f25
Oct 31 19:03:58.577165 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:58.577132    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:03:58.577242 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:58.577203    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:03:58.577242 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:58.577232    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:03:59.207356 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:59.207322    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:03:59.585502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:59.577958    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" event={"ID":"8d4422bb-55c9-4ba7-b963-850377365978","Type":"ContainerStarted","Data":"40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26"}
Oct 31 19:03:59.585502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:59.578077    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" event={"ID":"8d4422bb-55c9-4ba7-b963-850377365978","Type":"ContainerStarted","Data":"37756a208de787537b17435f7026924a9a7e39cc1f481162f517d31666fa3f25"}
Oct 31 19:03:59.699135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:03:59.699076    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" podStartSLOduration=2.699062103 podStartE2EDuration="2.699062103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:03:59.696648142 +0000 UTC m=+2585.843098008" watchObservedRunningTime="2026-10-31 19:03:59.699062103 +0000 UTC m=+2585.845511961"
Oct 31 19:04:00.143388 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:00.143354    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:04:00.144784 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:00.144756    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 31 19:04:00.144941 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:00.144805    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" podUID="8d4422bb-55c9-4ba7-b963-850377365978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 31 19:04:00.201801 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:00.201770    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:01.144006 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:01.143967    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 31 19:04:01.144553 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:01.144029    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" podUID="8d4422bb-55c9-4ba7-b963-850377365978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 31 19:04:02.143232 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:02.143192    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 31 19:04:02.143421 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:02.143273    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" podUID="8d4422bb-55c9-4ba7-b963-850377365978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 31 19:04:03.143194 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:03.143156    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 31 19:04:03.143645 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:03.143223    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" podUID="8d4422bb-55c9-4ba7-b963-850377365978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 31 19:04:04.143469 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:04.143432    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 31 19:04:04.143905 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:04.143530    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" podUID="8d4422bb-55c9-4ba7-b963-850377365978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 31 19:04:04.971665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:04.971623    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:05.143112 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:05.143074    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 31 19:04:05.143295 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:05.143143    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" podUID="8d4422bb-55c9-4ba7-b963-850377365978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 31 19:04:06.148381 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.148353    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:04:06.148793 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.148740    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:04:06.149595 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.149576    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:04:06.545436 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.545406    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"]
Oct 31 19:04:06.551030 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.551004    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:06.604182 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.604130    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:04:06.604182 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.604136    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:04:06.605467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.605440    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"]
Oct 31 19:04:06.727624 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.727595    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54f52fe0-r82g7\" (UID: \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:06.727788 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.727717    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54f52fe0-r82g7\" (UID: \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:06.727842 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.727805    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbpqc\" (UniqueName: \"kubernetes.io/projected/5af5672f-e6a1-4322-84d3-3577ac3fc55b-kube-api-access-nbpqc\") pod \"maas-api-revoke-sub-54f52fe0-r82g7\" (UID: \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:06.829157 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.829082    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbpqc\" (UniqueName: \"kubernetes.io/projected/5af5672f-e6a1-4322-84d3-3577ac3fc55b-kube-api-access-nbpqc\") pod \"maas-api-revoke-sub-54f52fe0-r82g7\" (UID: \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:06.829314 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.829196    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54f52fe0-r82g7\" (UID: \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:06.829314 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.829238    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54f52fe0-r82g7\" (UID: \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:06.829863 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.829835    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54f52fe0-r82g7\" (UID: \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:06.831782 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.831758    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54f52fe0-r82g7\" (UID: \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:06.886456 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:06.886424    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbpqc\" (UniqueName: \"kubernetes.io/projected/5af5672f-e6a1-4322-84d3-3577ac3fc55b-kube-api-access-nbpqc\") pod \"maas-api-revoke-sub-54f52fe0-r82g7\" (UID: \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:07.166091 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:07.166014    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:07.540318 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:07.540286    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"]
Oct 31 19:04:07.540501 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:04:07.540317    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af5672f_e6a1_4322_84d3_3577ac3fc55b.slice/crio-ba23406f3aae482733f10076bdc794b5f41689094bc0e5975ca27c1f66579313 WatchSource:0}: Error finding container ba23406f3aae482733f10076bdc794b5f41689094bc0e5975ca27c1f66579313: Status 404 returned error can't find the container with id ba23406f3aae482733f10076bdc794b5f41689094bc0e5975ca27c1f66579313
Oct 31 19:04:07.636097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:07.636068    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7" event={"ID":"5af5672f-e6a1-4322-84d3-3577ac3fc55b","Type":"ContainerStarted","Data":"ba23406f3aae482733f10076bdc794b5f41689094bc0e5975ca27c1f66579313"}
Oct 31 19:04:08.643848 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:08.643808    4484 generic.go:356] "Generic (PLEG): container finished" podID="5af5672f-e6a1-4322-84d3-3577ac3fc55b" containerID="0141515db88ca851f95ef477b52e8dded4a60a1dc50ee996db6f34a2d52ff7bc" exitCode=0
Oct 31 19:04:08.644341 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:08.643891    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7" event={"ID":"5af5672f-e6a1-4322-84d3-3577ac3fc55b","Type":"ContainerDied","Data":"0141515db88ca851f95ef477b52e8dded4a60a1dc50ee996db6f34a2d52ff7bc"}
Oct 31 19:04:09.766731 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:09.766696    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:09.804907 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:09.804872    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:09.968069 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:09.967984    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-xks-ca\") pod \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\" (UID: \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\") "
Oct 31 19:04:09.968069 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:09.968038    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af5672f-e6a1-4322-84d3-3577ac3fc55b-kube-api-access-nbpqc\" (UniqueName: \"kubernetes.io/projected/5af5672f-e6a1-4322-84d3-3577ac3fc55b-kube-api-access-nbpqc\") pod \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\" (UID: \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\") "
Oct 31 19:04:09.968311 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:09.968215    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-openshift-ca\") pod \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\" (UID: \"5af5672f-e6a1-4322-84d3-3577ac3fc55b\") "
Oct 31 19:04:09.968575 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:09.968550    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-openshift-ca" pod "5af5672f-e6a1-4322-84d3-3577ac3fc55b" (UID: "5af5672f-e6a1-4322-84d3-3577ac3fc55b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:04:09.968713 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:09.968694    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:04:09.970207 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:09.970177    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-xks-ca" pod "5af5672f-e6a1-4322-84d3-3577ac3fc55b" (UID: "5af5672f-e6a1-4322-84d3-3577ac3fc55b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:04:09.970368 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:09.970348    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af5672f-e6a1-4322-84d3-3577ac3fc55b-kube-api-access-nbpqc" pod "5af5672f-e6a1-4322-84d3-3577ac3fc55b" (UID: "5af5672f-e6a1-4322-84d3-3577ac3fc55b"). InnerVolumeSpecName "kube-api-access-nbpqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:04:10.069830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:10.069802    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af5672f-e6a1-4322-84d3-3577ac3fc55b-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:04:10.069830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:10.069827    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbpqc\" (UniqueName: \"kubernetes.io/projected/5af5672f-e6a1-4322-84d3-3577ac3fc55b-kube-api-access-nbpqc\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:04:10.657718 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:10.657690    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"
Oct 31 19:04:10.657905 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:10.657720    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7" event={"ID":"5af5672f-e6a1-4322-84d3-3577ac3fc55b","Type":"ContainerDied","Data":"ba23406f3aae482733f10076bdc794b5f41689094bc0e5975ca27c1f66579313"}
Oct 31 19:04:10.657905 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:10.657772    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba23406f3aae482733f10076bdc794b5f41689094bc0e5975ca27c1f66579313"
Oct 31 19:04:13.790789 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:13.790754    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:04:14.452269 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:14.452236    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:19.002408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:19.002375    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:19.201703 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:19.201665    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:20.029937 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.029903    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"]
Oct 31 19:04:20.030455 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.030442    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5af5672f-e6a1-4322-84d3-3577ac3fc55b" containerName="revoke-keys"
Oct 31 19:04:20.030548 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.030455    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af5672f-e6a1-4322-84d3-3577ac3fc55b" containerName="revoke-keys"
Oct 31 19:04:20.030590 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.030579    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af5672f-e6a1-4322-84d3-3577ac3fc55b" containerName="revoke-keys"
Oct 31 19:04:20.035007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.034990    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:20.083261 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.083224    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02c3bcac-v7wk6\" (UID: \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:20.083434 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.083273    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02c3bcac-v7wk6\" (UID: \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:20.083434 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.083300    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n9n9\" (UniqueName: \"kubernetes.io/projected/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-kube-api-access-6n9n9\") pod \"maas-api-revoke-sub-02c3bcac-v7wk6\" (UID: \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:20.089774 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.089748    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"]
Oct 31 19:04:20.136481 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.136448    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:04:20.136775 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.136758    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:04:20.184710 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.184682    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02c3bcac-v7wk6\" (UID: \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:20.184894 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.184724    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02c3bcac-v7wk6\" (UID: \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:20.184894 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.184741    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n9n9\" (UniqueName: \"kubernetes.io/projected/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-kube-api-access-6n9n9\") pod \"maas-api-revoke-sub-02c3bcac-v7wk6\" (UID: \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:20.185454 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.185427    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02c3bcac-v7wk6\" (UID: \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:20.187318 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.187292    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02c3bcac-v7wk6\" (UID: \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:20.242927 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.242891    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n9n9\" (UniqueName: \"kubernetes.io/projected/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-kube-api-access-6n9n9\") pod \"maas-api-revoke-sub-02c3bcac-v7wk6\" (UID: \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:20.348656 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.348574    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:20.720196 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:20.720170    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"]
Oct 31 19:04:20.722354 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:04:20.722313    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2e6eff_8e10_4f9a_94b0_baeb60a9eb83.slice/crio-c825154f977d4bd70940f42e23923cf17c10de56aa1ec3d73b93a65bff16031c WatchSource:0}: Error finding container c825154f977d4bd70940f42e23923cf17c10de56aa1ec3d73b93a65bff16031c: Status 404 returned error can't find the container with id c825154f977d4bd70940f42e23923cf17c10de56aa1ec3d73b93a65bff16031c
Oct 31 19:04:21.732578 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:21.732542    4484 generic.go:356] "Generic (PLEG): container finished" podID="5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83" containerID="6bd90dfaeccfdf84971c9eb52b23c0a24de60ec129d4a410e0880e186eeab4cc" exitCode=0
Oct 31 19:04:21.732995 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:21.732608    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6" event={"ID":"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83","Type":"ContainerDied","Data":"6bd90dfaeccfdf84971c9eb52b23c0a24de60ec129d4a410e0880e186eeab4cc"}
Oct 31 19:04:21.732995 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:21.732650    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6" event={"ID":"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83","Type":"ContainerStarted","Data":"c825154f977d4bd70940f42e23923cf17c10de56aa1ec3d73b93a65bff16031c"}
Oct 31 19:04:22.877184 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:22.877152    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:22.915784 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:22.915759    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-kube-api-access-6n9n9\" (UniqueName: \"kubernetes.io/projected/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-kube-api-access-6n9n9\") pod \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\" (UID: \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\") "
Oct 31 19:04:22.915945 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:22.915855    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-openshift-ca\") pod \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\" (UID: \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\") "
Oct 31 19:04:22.915994 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:22.915977    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-xks-ca\") pod \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\" (UID: \"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83\") "
Oct 31 19:04:22.916220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:22.916198    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-openshift-ca" pod "5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83" (UID: "5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:04:22.916319 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:22.916305    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:04:22.918033 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:22.918009    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-xks-ca" pod "5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83" (UID: "5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:04:22.918153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:22.918049    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-kube-api-access-6n9n9" pod "5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83" (UID: "5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83"). InnerVolumeSpecName "kube-api-access-6n9n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:04:23.017338 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:23.017243    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n9n9\" (UniqueName: \"kubernetes.io/projected/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-kube-api-access-6n9n9\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:04:23.017338 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:23.017271    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:04:23.746395 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:23.746365    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"
Oct 31 19:04:23.746581 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:23.746394    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6" event={"ID":"5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83","Type":"ContainerDied","Data":"c825154f977d4bd70940f42e23923cf17c10de56aa1ec3d73b93a65bff16031c"}
Oct 31 19:04:23.746581 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:23.746467    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c825154f977d4bd70940f42e23923cf17c10de56aa1ec3d73b93a65bff16031c"
Oct 31 19:04:23.981771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:23.981737    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:24.536638 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:24.536601    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:25.172384 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:25.172237    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:25.351928 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:25.351893    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:25.469354 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:04:25.469269    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:04:28.040230 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.040197    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"]
Oct 31 19:04:28.040783 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.040768    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83" containerName="revoke-keys"
Oct 31 19:04:28.040829 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.040784    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83" containerName="revoke-keys"
Oct 31 19:04:28.040900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.040891    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83" containerName="revoke-keys"
Oct 31 19:04:28.044016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.043996    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.136258 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.136222    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"]
Oct 31 19:04:28.185563 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.185526    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.185755 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.185596    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.185755 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.185697    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.185755 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.185728    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.185755 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.185751    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.185923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.185800    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86pr\" (UniqueName: \"kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-kube-api-access-w86pr\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.185923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.185866    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.185988 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.185952    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.185988 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.185981    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dd4393-2398-439f-aeac-8cb8abe74d77-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.287517 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.287460    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.287707 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.287545    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.287707 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.287569    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dd4393-2398-439f-aeac-8cb8abe74d77-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.287707 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.287590    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.287707 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.287634    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.287707 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.287700    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.287964 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.287724    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.287964 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.287798    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.287964 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.287823    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86pr\" (UniqueName: \"kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-kube-api-access-w86pr\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.288132 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.288000    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-credential-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.288192 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.288136    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-socket\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.288253 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.288232    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-certs\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.288351 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.288327    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-data\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.288466 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.288446    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dd4393-2398-439f-aeac-8cb8abe74d77-istiod-ca-cert\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.290082 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.290063    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-podinfo\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.290166 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.290145    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-envoy\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.345687 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.345611    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-token\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.345687 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.345670    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86pr\" (UniqueName: \"kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-kube-api-access-w86pr\") pod \"e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") " pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.357642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.357616    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:28.784580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.784535    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"]
Oct 31 19:04:28.822811 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:04:28.822783    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dd4393_2398_439f_aeac_8cb8abe74d77.slice/crio-fc709dd832535d9a6b4ca862dbf34e0821fd30001f344bdb9a0e846cfb91a165 WatchSource:0}: Error finding container fc709dd832535d9a6b4ca862dbf34e0821fd30001f344bdb9a0e846cfb91a165: Status 404 returned error can't find the container with id fc709dd832535d9a6b4ca862dbf34e0821fd30001f344bdb9a0e846cfb91a165
Oct 31 19:04:28.825253 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.825221    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:04:28.825343 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.825298    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:04:28.825343 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.825326    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:04:28.905389 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:28.905349    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:29.793229 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:29.793190    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6" event={"ID":"f8dd4393-2398-439f-aeac-8cb8abe74d77","Type":"ContainerStarted","Data":"adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce"}
Oct 31 19:04:29.793692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:29.793257    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6" event={"ID":"f8dd4393-2398-439f-aeac-8cb8abe74d77","Type":"ContainerStarted","Data":"fc709dd832535d9a6b4ca862dbf34e0821fd30001f344bdb9a0e846cfb91a165"}
Oct 31 19:04:29.884432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:29.884394    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:30.088105 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:30.087968    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:30.239241 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:30.239207    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:30.358385 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:30.358292    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:30.359693 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:30.359669    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 19:04:30.359831 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:30.359722    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6" podUID="f8dd4393-2398-439f-aeac-8cb8abe74d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 19:04:31.358404 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:31.358366    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 19:04:31.358828 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:31.358433    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6" podUID="f8dd4393-2398-439f-aeac-8cb8abe74d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 19:04:31.920866 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:31.920824    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:32.358050 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:32.358017    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 19:04:32.358322 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:32.358072    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6" podUID="f8dd4393-2398-439f-aeac-8cb8abe74d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 19:04:33.194095 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.194060    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:33.358309 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.358274    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 19:04:33.358512 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.358326    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6" podUID="f8dd4393-2398-439f-aeac-8cb8abe74d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 19:04:33.359283 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.359254    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:33.513205 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.513122    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:33.591085 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.591033    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6" podStartSLOduration=6.591017121 podStartE2EDuration="6.591017121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:29.983013545 +0000 UTC m=+2616.129463421" watchObservedRunningTime="2026-10-31 19:04:33.591017121 +0000 UTC m=+2619.737466985"
Oct 31 19:04:33.591724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.591705    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"]
Oct 31 19:04:33.595988 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.595971    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"
Oct 31 19:04:33.649155 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.649124    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-d8f34e-dockercfg-9f6s2\""
Oct 31 19:04:33.649610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.649589    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhsgv\" (UniqueName: \"kubernetes.io/projected/b3a0d3d9-238f-44e4-9894-11feed845de8-kube-api-access-vhsgv\") pod \"payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2\" (UID: \"b3a0d3d9-238f-44e4-9894-11feed845de8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"
Oct 31 19:04:33.649681 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.649632    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a0d3d9-238f-44e4-9894-11feed845de8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2\" (UID: \"b3a0d3d9-238f-44e4-9894-11feed845de8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"
Oct 31 19:04:33.653140 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.653113    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"]
Oct 31 19:04:33.693334 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.693289    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:33.699652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.699621    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-d8f34e\""
Oct 31 19:04:33.733567 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.733526    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"]
Oct 31 19:04:33.739502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.738661    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"
Oct 31 19:04:33.750548 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.750523    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhsgv\" (UniqueName: \"kubernetes.io/projected/b3a0d3d9-238f-44e4-9894-11feed845de8-kube-api-access-vhsgv\") pod \"payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2\" (UID: \"b3a0d3d9-238f-44e4-9894-11feed845de8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"
Oct 31 19:04:33.750766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.750580    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a0d3d9-238f-44e4-9894-11feed845de8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2\" (UID: \"b3a0d3d9-238f-44e4-9894-11feed845de8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"
Oct 31 19:04:33.751433 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.751411    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a0d3d9-238f-44e4-9894-11feed845de8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2\" (UID: \"b3a0d3d9-238f-44e4-9894-11feed845de8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"
Oct 31 19:04:33.795316 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.795221    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"]
Oct 31 19:04:33.808796 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.808767    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhsgv\" (UniqueName: \"kubernetes.io/projected/b3a0d3d9-238f-44e4-9894-11feed845de8-kube-api-access-vhsgv\") pod \"payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2\" (UID: \"b3a0d3d9-238f-44e4-9894-11feed845de8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"
Oct 31 19:04:33.851432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.851402    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzqs7\" (UniqueName: \"kubernetes.io/projected/5410d4d5-bc30-456d-a17b-65e13b084937-kube-api-access-xzqs7\") pod \"payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq\" (UID: \"5410d4d5-bc30-456d-a17b-65e13b084937\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"
Oct 31 19:04:33.851662 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.851633    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5410d4d5-bc30-456d-a17b-65e13b084937-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq\" (UID: \"5410d4d5-bc30-456d-a17b-65e13b084937\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"
Oct 31 19:04:33.913432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.913399    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"
Oct 31 19:04:33.953305 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.953232    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5410d4d5-bc30-456d-a17b-65e13b084937-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq\" (UID: \"5410d4d5-bc30-456d-a17b-65e13b084937\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"
Oct 31 19:04:33.953515 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.953339    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzqs7\" (UniqueName: \"kubernetes.io/projected/5410d4d5-bc30-456d-a17b-65e13b084937-kube-api-access-xzqs7\") pod \"payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq\" (UID: \"5410d4d5-bc30-456d-a17b-65e13b084937\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"
Oct 31 19:04:33.954653 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:33.954013    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5410d4d5-bc30-456d-a17b-65e13b084937-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq\" (UID: \"5410d4d5-bc30-456d-a17b-65e13b084937\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"
Oct 31 19:04:34.016097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.013364    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzqs7\" (UniqueName: \"kubernetes.io/projected/5410d4d5-bc30-456d-a17b-65e13b084937-kube-api-access-xzqs7\") pod \"payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq\" (UID: \"5410d4d5-bc30-456d-a17b-65e13b084937\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"
Oct 31 19:04:34.057297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.057217    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"
Oct 31 19:04:34.311426 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.311397    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"]
Oct 31 19:04:34.311923 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:04:34.311902    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a0d3d9_238f_44e4_9894_11feed845de8.slice/crio-ce9f2bf67c7930b8a42b6494f54e4e6e2efdfed3071a1fdf8f7a3fa36ac7571a WatchSource:0}: Error finding container ce9f2bf67c7930b8a42b6494f54e4e6e2efdfed3071a1fdf8f7a3fa36ac7571a: Status 404 returned error can't find the container with id ce9f2bf67c7930b8a42b6494f54e4e6e2efdfed3071a1fdf8f7a3fa36ac7571a
Oct 31 19:04:34.353005 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.352982    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:34.358102 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.358082    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 19:04:34.358180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.358123    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6" podUID="f8dd4393-2398-439f-aeac-8cb8abe74d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 19:04:34.455417 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.455393    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"]
Oct 31 19:04:34.456027 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:04:34.456000    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5410d4d5_bc30_456d_a17b_65e13b084937.slice/crio-faf113f1bfc568c4a376955a111046a5bedc7ead54e234a4ab1212f0ca7b22c1 WatchSource:0}: Error finding container faf113f1bfc568c4a376955a111046a5bedc7ead54e234a4ab1212f0ca7b22c1: Status 404 returned error can't find the container with id faf113f1bfc568c4a376955a111046a5bedc7ead54e234a4ab1212f0ca7b22c1
Oct 31 19:04:34.828115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.828079    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2" event={"ID":"b3a0d3d9-238f-44e4-9894-11feed845de8","Type":"ContainerStarted","Data":"6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36"}
Oct 31 19:04:34.828294 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.828134    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2" event={"ID":"b3a0d3d9-238f-44e4-9894-11feed845de8","Type":"ContainerStarted","Data":"ce9f2bf67c7930b8a42b6494f54e4e6e2efdfed3071a1fdf8f7a3fa36ac7571a"}
Oct 31 19:04:34.828294 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.828209    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"
Oct 31 19:04:34.829842 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.829812    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq" event={"ID":"5410d4d5-bc30-456d-a17b-65e13b084937","Type":"ContainerStarted","Data":"0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212"}
Oct 31 19:04:34.829964 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.829846    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq" event={"ID":"5410d4d5-bc30-456d-a17b-65e13b084937","Type":"ContainerStarted","Data":"faf113f1bfc568c4a376955a111046a5bedc7ead54e234a4ab1212f0ca7b22c1"}
Oct 31 19:04:34.829964 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.829873    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"
Oct 31 19:04:34.945452 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:34.945401    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2" podStartSLOduration=1.945387304 podStartE2EDuration="1.945387304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:34.942090519 +0000 UTC m=+2621.088540383" watchObservedRunningTime="2026-10-31 19:04:34.945387304 +0000 UTC m=+2621.091837167"
Oct 31 19:04:35.056998 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:35.056947    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq" podStartSLOduration=2.056930784 podStartE2EDuration="2.056930784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:35.055396598 +0000 UTC m=+2621.201846464" watchObservedRunningTime="2026-10-31 19:04:35.056930784 +0000 UTC m=+2621.203380649"
Oct 31 19:04:35.358579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:35.358543    4484 patch_prober.go:29] interesting pod/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 19:04:35.358979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:35.358602    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6" podUID="f8dd4393-2398-439f-aeac-8cb8abe74d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 19:04:36.363104 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:36.363076    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:36.363523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:36.363500    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:36.364180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:36.364161    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:04:37.891437 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:37.891402    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:38.055423 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:38.055384    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:38.223319 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:38.223236    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:39.674979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:39.674945    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:42.609689 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:42.609655    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:44.573225 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:44.573183    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:44.819978 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:44.819946    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:44.979252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:44.979172    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:45.127976 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.127939    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:45.310524 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.310441    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"]
Oct 31 19:04:45.326769 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.326738    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"]
Oct 31 19:04:45.327091 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.327075    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"
Oct 31 19:04:45.369927 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.369901    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"]
Oct 31 19:04:45.373745 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.373728    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"
Oct 31 19:04:45.382097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.382077    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-bbd8c6-dockercfg-89ll9\""
Oct 31 19:04:45.382228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.382158    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-bbd8c6\""
Oct 31 19:04:45.399418 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.399391    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd9lf\" (UniqueName: \"kubernetes.io/projected/169ec18f-05f8-4fec-a540-9ade4b14863d-kube-api-access-qd9lf\") pod \"payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq\" (UID: \"169ec18f-05f8-4fec-a540-9ade4b14863d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"
Oct 31 19:04:45.399636 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.399617    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c010d33f-79f2-4396-83aa-ef85d00d4c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx\" (UID: \"c010d33f-79f2-4396-83aa-ef85d00d4c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"
Oct 31 19:04:45.399715 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.399680    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169ec18f-05f8-4fec-a540-9ade4b14863d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq\" (UID: \"169ec18f-05f8-4fec-a540-9ade4b14863d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"
Oct 31 19:04:45.399760 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.399744    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr68g\" (UniqueName: \"kubernetes.io/projected/c010d33f-79f2-4396-83aa-ef85d00d4c81-kube-api-access-rr68g\") pod \"payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx\" (UID: \"c010d33f-79f2-4396-83aa-ef85d00d4c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"
Oct 31 19:04:45.426243 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.426213    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"]
Oct 31 19:04:45.501029 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.500993    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c010d33f-79f2-4396-83aa-ef85d00d4c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx\" (UID: \"c010d33f-79f2-4396-83aa-ef85d00d4c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"
Oct 31 19:04:45.501217 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.501048    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169ec18f-05f8-4fec-a540-9ade4b14863d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq\" (UID: \"169ec18f-05f8-4fec-a540-9ade4b14863d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"
Oct 31 19:04:45.501217 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.501083    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr68g\" (UniqueName: \"kubernetes.io/projected/c010d33f-79f2-4396-83aa-ef85d00d4c81-kube-api-access-rr68g\") pod \"payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx\" (UID: \"c010d33f-79f2-4396-83aa-ef85d00d4c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"
Oct 31 19:04:45.501217 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.501107    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd9lf\" (UniqueName: \"kubernetes.io/projected/169ec18f-05f8-4fec-a540-9ade4b14863d-kube-api-access-qd9lf\") pod \"payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq\" (UID: \"169ec18f-05f8-4fec-a540-9ade4b14863d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"
Oct 31 19:04:45.501953 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.501930    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c010d33f-79f2-4396-83aa-ef85d00d4c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx\" (UID: \"c010d33f-79f2-4396-83aa-ef85d00d4c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"
Oct 31 19:04:45.502004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.501930    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169ec18f-05f8-4fec-a540-9ade4b14863d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq\" (UID: \"169ec18f-05f8-4fec-a540-9ade4b14863d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"
Oct 31 19:04:45.562335 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.562268    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd9lf\" (UniqueName: \"kubernetes.io/projected/169ec18f-05f8-4fec-a540-9ade4b14863d-kube-api-access-qd9lf\") pod \"payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq\" (UID: \"169ec18f-05f8-4fec-a540-9ade4b14863d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"
Oct 31 19:04:45.562505 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.562457    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr68g\" (UniqueName: \"kubernetes.io/projected/c010d33f-79f2-4396-83aa-ef85d00d4c81-kube-api-access-rr68g\") pod \"payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx\" (UID: \"c010d33f-79f2-4396-83aa-ef85d00d4c81\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"
Oct 31 19:04:45.662119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.662083    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"
Oct 31 19:04:45.687627 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.687586    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"
Oct 31 19:04:45.840560 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.840459    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"
Oct 31 19:04:45.840560 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.840549    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"
Oct 31 19:04:45.898355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:45.898320    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:46.073576 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:46.073547    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"]
Oct 31 19:04:46.073804 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:04:46.073783    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc010d33f_79f2_4396_83aa_ef85d00d4c81.slice/crio-94ef2fba7847eacb37c16dd3ee5f53be16c2306cf9f31403cab14728de1fc759 WatchSource:0}: Error finding container 94ef2fba7847eacb37c16dd3ee5f53be16c2306cf9f31403cab14728de1fc759: Status 404 returned error can't find the container with id 94ef2fba7847eacb37c16dd3ee5f53be16c2306cf9f31403cab14728de1fc759
Oct 31 19:04:46.118254 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:46.118225    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"]
Oct 31 19:04:46.118641 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:04:46.118616    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169ec18f_05f8_4fec_a540_9ade4b14863d.slice/crio-404c0ef1ff7324eed2255cef18f8b45385f138fe70eb427a7ee1cd11e15444da WatchSource:0}: Error finding container 404c0ef1ff7324eed2255cef18f8b45385f138fe70eb427a7ee1cd11e15444da: Status 404 returned error can't find the container with id 404c0ef1ff7324eed2255cef18f8b45385f138fe70eb427a7ee1cd11e15444da
Oct 31 19:04:46.747216 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:46.747182    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:46.921328 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:46.921291    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:46.925237 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:46.925210    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq" event={"ID":"169ec18f-05f8-4fec-a540-9ade4b14863d","Type":"ContainerStarted","Data":"6dd354e23edafee9517db07e36a0c29735df9ed94ae817d56898ea0bb5b9c25c"}
Oct 31 19:04:46.925421 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:46.925259    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq" event={"ID":"169ec18f-05f8-4fec-a540-9ade4b14863d","Type":"ContainerStarted","Data":"404c0ef1ff7324eed2255cef18f8b45385f138fe70eb427a7ee1cd11e15444da"}
Oct 31 19:04:46.925421 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:46.925298    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"
Oct 31 19:04:46.926926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:46.926898    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx" event={"ID":"c010d33f-79f2-4396-83aa-ef85d00d4c81","Type":"ContainerStarted","Data":"d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22"}
Oct 31 19:04:46.927043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:46.926943    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx" event={"ID":"c010d33f-79f2-4396-83aa-ef85d00d4c81","Type":"ContainerStarted","Data":"94ef2fba7847eacb37c16dd3ee5f53be16c2306cf9f31403cab14728de1fc759"}
Oct 31 19:04:46.927043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:46.926983    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"
Oct 31 19:04:47.041743 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:47.041645    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq" podStartSLOduration=2.041628139 podStartE2EDuration="2.041628139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:47.039970817 +0000 UTC m=+2633.186420680" watchObservedRunningTime="2026-10-31 19:04:47.041628139 +0000 UTC m=+2633.188078003"
Oct 31 19:04:47.085345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:47.085291    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:47.155863 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:47.155815    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx" podStartSLOduration=2.155801719 podStartE2EDuration="2.155801719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:47.153740008 +0000 UTC m=+2633.300189872" watchObservedRunningTime="2026-10-31 19:04:47.155801719 +0000 UTC m=+2633.302251620"
Oct 31 19:04:49.390852 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:49.390816    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:50.632324 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:50.632264    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:51.133381 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:51.133342    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:51.232840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:51.232804    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"]
Oct 31 19:04:51.353836 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:51.353805    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:51.619916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:51.619885    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:51.787046 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:51.787002    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:54.654549 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:54.654513    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:55.001504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.001460    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:55.156028 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.155990    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:55.298803 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.298710    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:55.393982 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.393951    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"]
Oct 31 19:04:55.404304 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.404247    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"
Oct 31 19:04:55.453673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.453638    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"]
Oct 31 19:04:55.495535 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:04:55.490900    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:04:55.499787 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.499535    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-dbceb9-dockercfg-zqj24\""
Oct 31 19:04:55.499787 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.499587    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-dbceb9\""
Oct 31 19:04:55.528596 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.528566    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e06783-c3be-4a94-99d1-a60f996d6044-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl\" (UID: \"30e06783-c3be-4a94-99d1-a60f996d6044\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"
Oct 31 19:04:55.528754 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.528606    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdsz\" (UniqueName: \"kubernetes.io/projected/30e06783-c3be-4a94-99d1-a60f996d6044-kube-api-access-8hdsz\") pod \"payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl\" (UID: \"30e06783-c3be-4a94-99d1-a60f996d6044\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"
Oct 31 19:04:55.629616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.629529    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e06783-c3be-4a94-99d1-a60f996d6044-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl\" (UID: \"30e06783-c3be-4a94-99d1-a60f996d6044\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"
Oct 31 19:04:55.629616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.629576    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdsz\" (UniqueName: \"kubernetes.io/projected/30e06783-c3be-4a94-99d1-a60f996d6044-kube-api-access-8hdsz\") pod \"payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl\" (UID: \"30e06783-c3be-4a94-99d1-a60f996d6044\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"
Oct 31 19:04:55.630355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.630325    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e06783-c3be-4a94-99d1-a60f996d6044-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl\" (UID: \"30e06783-c3be-4a94-99d1-a60f996d6044\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"
Oct 31 19:04:55.687088 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.687060    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdsz\" (UniqueName: \"kubernetes.io/projected/30e06783-c3be-4a94-99d1-a60f996d6044-kube-api-access-8hdsz\") pod \"payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl\" (UID: \"30e06783-c3be-4a94-99d1-a60f996d6044\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"
Oct 31 19:04:55.720915 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:55.720873    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"
Oct 31 19:04:56.093574 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.093543    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"]
Oct 31 19:04:56.093721 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:04:56.093692    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e06783_c3be_4a94_99d1_a60f996d6044.slice/crio-57dd89a6be68bfc721463966568847847576f66e3c5ceb3e88055213f0e05e14 WatchSource:0}: Error finding container 57dd89a6be68bfc721463966568847847576f66e3c5ceb3e88055213f0e05e14: Status 404 returned error can't find the container with id 57dd89a6be68bfc721463966568847847576f66e3c5ceb3e88055213f0e05e14
Oct 31 19:04:56.233169 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.233130    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d" podUID="3acf824f-751d-4b09-80ad-c1f88b785502" containerName="payload-processing" containerID="cri-o://4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a" gracePeriod=25
Oct 31 19:04:56.466492 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.466443    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:56.486348 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.486325    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"
Oct 31 19:04:56.540035 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.540000    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3acf824f-751d-4b09-80ad-c1f88b785502-kube-api-access-gpkxk\" (UniqueName: \"kubernetes.io/projected/3acf824f-751d-4b09-80ad-c1f88b785502-kube-api-access-gpkxk\") pod \"3acf824f-751d-4b09-80ad-c1f88b785502\" (UID: \"3acf824f-751d-4b09-80ad-c1f88b785502\") "
Oct 31 19:04:56.540228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.540070    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3acf824f-751d-4b09-80ad-c1f88b785502-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3acf824f-751d-4b09-80ad-c1f88b785502-plugins-config-volume\") pod \"3acf824f-751d-4b09-80ad-c1f88b785502\" (UID: \"3acf824f-751d-4b09-80ad-c1f88b785502\") "
Oct 31 19:04:56.540463 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.540438    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acf824f-751d-4b09-80ad-c1f88b785502-plugins-config-volume" pod "3acf824f-751d-4b09-80ad-c1f88b785502" (UID: "3acf824f-751d-4b09-80ad-c1f88b785502"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:04:56.542228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.542207    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acf824f-751d-4b09-80ad-c1f88b785502-kube-api-access-gpkxk" pod "3acf824f-751d-4b09-80ad-c1f88b785502" (UID: "3acf824f-751d-4b09-80ad-c1f88b785502"). InnerVolumeSpecName "kube-api-access-gpkxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:04:56.641673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.641592    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpkxk\" (UniqueName: \"kubernetes.io/projected/3acf824f-751d-4b09-80ad-c1f88b785502-kube-api-access-gpkxk\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:04:56.641673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.641629    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3acf824f-751d-4b09-80ad-c1f88b785502-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:04:56.992451 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.992413    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl" event={"ID":"30e06783-c3be-4a94-99d1-a60f996d6044","Type":"ContainerStarted","Data":"b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998"}
Oct 31 19:04:56.992951 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.992494    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"
Oct 31 19:04:56.992951 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.992515    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl" event={"ID":"30e06783-c3be-4a94-99d1-a60f996d6044","Type":"ContainerStarted","Data":"57dd89a6be68bfc721463966568847847576f66e3c5ceb3e88055213f0e05e14"}
Oct 31 19:04:56.993931 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.993907    4484 generic.go:356] "Generic (PLEG): container finished" podID="3acf824f-751d-4b09-80ad-c1f88b785502" containerID="4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a" exitCode=0
Oct 31 19:04:56.994060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.993940    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"
Oct 31 19:04:56.994060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.993966    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d" event={"ID":"3acf824f-751d-4b09-80ad-c1f88b785502","Type":"ContainerDied","Data":"4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a"}
Oct 31 19:04:56.994060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.994010    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d" event={"ID":"3acf824f-751d-4b09-80ad-c1f88b785502","Type":"ContainerDied","Data":"677aa78102c56b47e03de18661e4376903cbbebaf67d4dd964fd877be88928c7"}
Oct 31 19:04:56.994060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:56.994038    4484 scope.go:122] "RemoveContainer" containerID="4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a"
Oct 31 19:04:57.004464 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:57.004446    4484 scope.go:122] "RemoveContainer" containerID="4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a"
Oct 31 19:04:57.004752 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:04:57.004723    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a\": container with ID starting with 4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a not found: ID does not exist" containerID="4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a"
Oct 31 19:04:57.004814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:57.004752    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a"} err="failed to get container status \"4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a\": rpc error: code = NotFound desc = could not find container \"4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a\": container with ID starting with 4ccc8682f2fc4d2d0c5db207f94a110648df2800ce231a5d316f4fca1de0db3a not found: ID does not exist"
Oct 31 19:04:57.109008 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:57.108955    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl" podStartSLOduration=2.108940384 podStartE2EDuration="2.108940384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:04:57.106747198 +0000 UTC m=+2643.253197060" watchObservedRunningTime="2026-10-31 19:04:57.108940384 +0000 UTC m=+2643.255390247"
Oct 31 19:04:57.219372 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:57.219332    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"]
Oct 31 19:04:57.222216 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:57.222189    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-79e7b6-7d667b9747-tln4d"]
Oct 31 19:04:57.937733 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:57.937700    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"
Oct 31 19:04:57.938015 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:57.937994    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"
Oct 31 19:04:57.993894 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:57.993864    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3acf824f-751d-4b09-80ad-c1f88b785502" path="/var/lib/kubelet/pods/3acf824f-751d-4b09-80ad-c1f88b785502/volumes"
Oct 31 19:04:59.530701 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:59.530663    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:04:59.679557 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:04:59.679519    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:02.069860 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:02.069826    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:05:04.305279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:04.305239    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:08.003772 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:08.003687    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"
Oct 31 19:05:08.352089 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:08.351998    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:08.983655 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:08.983626    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"]
Oct 31 19:05:08.984237 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:08.984223    4484 cpu_manager.go:414] "containerMap: removing container" podUID="3acf824f-751d-4b09-80ad-c1f88b785502" containerName="payload-processing"
Oct 31 19:05:08.984283 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:08.984239    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3acf824f-751d-4b09-80ad-c1f88b785502" containerName="payload-processing"
Oct 31 19:05:08.984336 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:08.984327    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3acf824f-751d-4b09-80ad-c1f88b785502" containerName="payload-processing"
Oct 31 19:05:08.989247 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:08.989228    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.087282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.087246    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.087794 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.087385    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471772d0-1583-42bb-83c1-0858499b7210-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.087794 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.087415    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471772d0-1583-42bb-83c1-0858499b7210-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.087794 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.087599    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.087794 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.087648    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.087794 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.087789    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.088067 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.087827    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.088067 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.087858    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w6ff\" (UniqueName: \"kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-kube-api-access-5w6ff\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.088067 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.087919    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.094528 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.094499    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"]
Oct 31 19:05:09.189504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.188776    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.189504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.188835    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.189504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.188869    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w6ff\" (UniqueName: \"kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-kube-api-access-5w6ff\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.189504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.188914    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.189504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.188943    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.189504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.189019    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471772d0-1583-42bb-83c1-0858499b7210-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.189504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.189042    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471772d0-1583-42bb-83c1-0858499b7210-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.189504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.189167    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.189504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.189199    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.190254 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.189739    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.190254 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.189970    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-certs\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.190254 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.190182    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-credential-socket\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.197197 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.190748    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-data\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.197197 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.194012    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471772d0-1583-42bb-83c1-0858499b7210-istiod-ca-cert\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.211359 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.211325    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471772d0-1583-42bb-83c1-0858499b7210-istio-podinfo\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.213303 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.213071    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-envoy\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.247747 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.247708    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-istio-token\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.247747 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.247732    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w6ff\" (UniqueName: \"kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-kube-api-access-5w6ff\") pod \"e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") " pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.302545 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.302514    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:09.729904 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.729875    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"]
Oct 31 19:05:09.730444 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:05:09.730416    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471772d0_1583_42bb_83c1_0858499b7210.slice/crio-fb339a121272af0ba594d73e00c2aaf2e0f900a090dfb30acd00d5a6dbea2767 WatchSource:0}: Error finding container fb339a121272af0ba594d73e00c2aaf2e0f900a090dfb30acd00d5a6dbea2767: Status 404 returned error can't find the container with id fb339a121272af0ba594d73e00c2aaf2e0f900a090dfb30acd00d5a6dbea2767
Oct 31 19:05:09.732800 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.732766    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:05:09.732883 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.732830    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:05:09.732883 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:09.732861    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:05:10.045132 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:10.045094    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:10.080513 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:10.080455    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" event={"ID":"471772d0-1583-42bb-83c1-0858499b7210","Type":"ContainerStarted","Data":"3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb"}
Oct 31 19:05:10.080677 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:10.080535    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" event={"ID":"471772d0-1583-42bb-83c1-0858499b7210","Type":"ContainerStarted","Data":"fb339a121272af0ba594d73e00c2aaf2e0f900a090dfb30acd00d5a6dbea2767"}
Oct 31 19:05:10.198555 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:10.198506    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" podStartSLOduration=2.19846183 podStartE2EDuration="2.19846183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:05:10.197843486 +0000 UTC m=+2656.344293351" watchObservedRunningTime="2026-10-31 19:05:10.19846183 +0000 UTC m=+2656.344911693"
Oct 31 19:05:10.285501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:10.276630    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:10.303457 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:10.303378    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:10.305223 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:10.305198    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:05:10.305371 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:10.305240    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" podUID="471772d0-1583-42bb-83c1-0858499b7210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:05:10.436874 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:10.436839    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:10.941599 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:10.941563    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:11.142732 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:11.142688    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:11.303190 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:11.303152    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:05:11.303619 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:11.303219    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" podUID="471772d0-1583-42bb-83c1-0858499b7210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:05:11.924120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:11.924084    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:12.303118 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:12.303080    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:05:12.303289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:12.303137    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" podUID="471772d0-1583-42bb-83c1-0858499b7210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:05:12.435153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:12.435106    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:12.987247 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:12.987216    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:13.142846 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.142814    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:13.303936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.303846    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:05:13.303936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.303915    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" podUID="471772d0-1583-42bb-83c1-0858499b7210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:05:13.697299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.697208    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"]
Oct 31 19:05:13.701570 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.701549    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.747171 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.747135    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-envoy\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.747355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.747249    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-credential-socket\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.747355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.747283    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbxvj\" (UniqueName: \"kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-kube-api-access-jbxvj\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.747503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.747461    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-data\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.747565 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.747545    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-token\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.747611 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.747596    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istiod-ca-cert\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.747724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.747707    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-socket\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.747759 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.747749    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-podinfo\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.747801 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.747787    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-certs\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.758537 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.754380    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7006d3-openshift-default-dockercfg-kx9rz\""
Oct 31 19:05:13.808123 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.808093    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"]
Oct 31 19:05:13.828164 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.828105    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:13.848432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.848397    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-credential-socket\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.848654 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.848436    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbxvj\" (UniqueName: \"kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-kube-api-access-jbxvj\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.848654 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.848514    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-data\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.848654 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.848538    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-token\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.848841 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.848646    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istiod-ca-cert\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.848841 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.848757    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-socket\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.848841 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.848788    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-podinfo\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.848986 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.848903    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-credential-socket\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.848986 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.848919    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-certs\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.848986 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.848939    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-data\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.849079 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.849017    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-envoy\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.849125 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.849075    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-socket\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.849197 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.849169    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-certs\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.849332 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.849318    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istiod-ca-cert\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.851320 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.851297    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-envoy\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.851525 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.851500    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-podinfo\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.909541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.909513    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-token\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:13.909711 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:13.909590    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbxvj\" (UniqueName: \"kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-kube-api-access-jbxvj\") pod \"e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") " pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:14.015870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:14.015834    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:14.029215 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:14.029183    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:14.302892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:14.302814    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:05:14.303067 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:14.302886    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" podUID="471772d0-1583-42bb-83c1-0858499b7210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:05:14.487521 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:14.487493    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"]
Oct 31 19:05:14.489043 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:05:14.489016    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5113a87a_ad1d_4ba0_b056_d03fd4fd6a0d.slice/crio-8b9fd76058ec3d08737ea930ecb88f768ad0103efaf5c387e690e8739ac24202 WatchSource:0}: Error finding container 8b9fd76058ec3d08737ea930ecb88f768ad0103efaf5c387e690e8739ac24202: Status 404 returned error can't find the container with id 8b9fd76058ec3d08737ea930ecb88f768ad0103efaf5c387e690e8739ac24202
Oct 31 19:05:14.492165 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:14.492134    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:05:14.492249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:14.492211    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:05:14.492249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:14.492241    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:05:15.126978 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:15.126941    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" event={"ID":"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d","Type":"ContainerStarted","Data":"4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7"}
Oct 31 19:05:15.127166 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:15.126996    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" event={"ID":"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d","Type":"ContainerStarted","Data":"8b9fd76058ec3d08737ea930ecb88f768ad0103efaf5c387e690e8739ac24202"}
Oct 31 19:05:15.251814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:15.251763    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" podStartSLOduration=2.251748011 podStartE2EDuration="2.251748011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:05:15.249905049 +0000 UTC m=+2661.396354912" watchObservedRunningTime="2026-10-31 19:05:15.251748011 +0000 UTC m=+2661.398197933"
Oct 31 19:05:15.303285 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:15.303249    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:05:15.303448 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:15.303322    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" podUID="471772d0-1583-42bb-83c1-0858499b7210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:05:15.482558 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:15.480466    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:16.016781 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:16.016749    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:16.022457 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:16.022433    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:16.133302 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:16.133262    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:16.134346 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:16.134325    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:05:16.303095 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:16.303008    4484 patch_prober.go:29] interesting pod/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:05:16.303095 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:16.303071    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" podUID="471772d0-1583-42bb-83c1-0858499b7210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:05:17.307770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:17.307741    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:17.308160 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:17.308136    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:17.308810 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:17.308790    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:05:18.020142 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:18.020108    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:18.182205 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:18.182172    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:18.336120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:18.336028    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:18.500233 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:18.500195    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:20.489806 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:20.489761    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:21.199389 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:21.199355    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:21.396421 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:21.396381    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:24.486780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:24.486744    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:24.611355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:24.611322    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"]
Oct 31 19:05:24.718066 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:24.718036    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"]
Oct 31 19:05:24.992443 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:24.992413    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:25.139904 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:25.139870    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:25.463001 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:25.462907    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:25.469099 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:05:25.469069    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:05:25.611080 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:25.611040    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:25.772658 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:25.772622    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:29.611793 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:29.611750    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9" podUID="8056bf6d-b304-4656-af4c-8a5b8708845b" containerName="payload-processing" containerID="cri-o://c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680" gracePeriod=25
Oct 31 19:05:29.718887 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:29.718843    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b" podUID="9fe44e72-b397-4171-9952-56f02961c069" containerName="payload-pre-processing" containerID="cri-o://c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e" gracePeriod=25
Oct 31 19:05:29.888628 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:29.888599    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"
Oct 31 19:05:29.968664 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:29.968611    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8056bf6d-b304-4656-af4c-8a5b8708845b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8056bf6d-b304-4656-af4c-8a5b8708845b-plugins-config-volume\") pod \"8056bf6d-b304-4656-af4c-8a5b8708845b\" (UID: \"8056bf6d-b304-4656-af4c-8a5b8708845b\") "
Oct 31 19:05:29.968859 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:29.968835    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8056bf6d-b304-4656-af4c-8a5b8708845b-kube-api-access-q6x94\" (UniqueName: \"kubernetes.io/projected/8056bf6d-b304-4656-af4c-8a5b8708845b-kube-api-access-q6x94\") pod \"8056bf6d-b304-4656-af4c-8a5b8708845b\" (UID: \"8056bf6d-b304-4656-af4c-8a5b8708845b\") "
Oct 31 19:05:29.968967 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:29.968946    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8056bf6d-b304-4656-af4c-8a5b8708845b-plugins-config-volume" pod "8056bf6d-b304-4656-af4c-8a5b8708845b" (UID: "8056bf6d-b304-4656-af4c-8a5b8708845b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:05:29.969356 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:29.969341    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8056bf6d-b304-4656-af4c-8a5b8708845b-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:05:29.971026 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:29.971000    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8056bf6d-b304-4656-af4c-8a5b8708845b-kube-api-access-q6x94" pod "8056bf6d-b304-4656-af4c-8a5b8708845b" (UID: "8056bf6d-b304-4656-af4c-8a5b8708845b"). InnerVolumeSpecName "kube-api-access-q6x94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:05:29.989691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:29.989660    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"
Oct 31 19:05:30.070254 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.070216    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe44e72-b397-4171-9952-56f02961c069-kube-api-access-b6t8n\" (UniqueName: \"kubernetes.io/projected/9fe44e72-b397-4171-9952-56f02961c069-kube-api-access-b6t8n\") pod \"9fe44e72-b397-4171-9952-56f02961c069\" (UID: \"9fe44e72-b397-4171-9952-56f02961c069\") "
Oct 31 19:05:30.070544 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.070526    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe44e72-b397-4171-9952-56f02961c069-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe44e72-b397-4171-9952-56f02961c069-plugins-config-volume\") pod \"9fe44e72-b397-4171-9952-56f02961c069\" (UID: \"9fe44e72-b397-4171-9952-56f02961c069\") "
Oct 31 19:05:30.070874 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.070851    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe44e72-b397-4171-9952-56f02961c069-plugins-config-volume" pod "9fe44e72-b397-4171-9952-56f02961c069" (UID: "9fe44e72-b397-4171-9952-56f02961c069"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:05:30.071080 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.071063    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6x94\" (UniqueName: \"kubernetes.io/projected/8056bf6d-b304-4656-af4c-8a5b8708845b-kube-api-access-q6x94\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:05:30.071152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.071086    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fe44e72-b397-4171-9952-56f02961c069-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:05:30.072622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.072599    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe44e72-b397-4171-9952-56f02961c069-kube-api-access-b6t8n" pod "9fe44e72-b397-4171-9952-56f02961c069" (UID: "9fe44e72-b397-4171-9952-56f02961c069"). InnerVolumeSpecName "kube-api-access-b6t8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:05:30.084772 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.084750    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:30.172178 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.172091    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6t8n\" (UniqueName: \"kubernetes.io/projected/9fe44e72-b397-4171-9952-56f02961c069-kube-api-access-b6t8n\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:05:30.226996 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.226962    4484 generic.go:356] "Generic (PLEG): container finished" podID="8056bf6d-b304-4656-af4c-8a5b8708845b" containerID="c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680" exitCode=0
Oct 31 19:05:30.227161 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.227002    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"
Oct 31 19:05:30.227161 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.227047    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9" event={"ID":"8056bf6d-b304-4656-af4c-8a5b8708845b","Type":"ContainerDied","Data":"c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680"}
Oct 31 19:05:30.227161 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.227102    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9" event={"ID":"8056bf6d-b304-4656-af4c-8a5b8708845b","Type":"ContainerDied","Data":"c514882708f5468071130bd04cf52ff2d8faa3e5f478da29895707117a157b19"}
Oct 31 19:05:30.227161 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.227131    4484 scope.go:122] "RemoveContainer" containerID="c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680"
Oct 31 19:05:30.228634 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.228612    4484 generic.go:356] "Generic (PLEG): container finished" podID="9fe44e72-b397-4171-9952-56f02961c069" containerID="c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e" exitCode=0
Oct 31 19:05:30.228757 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.228647    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b" event={"ID":"9fe44e72-b397-4171-9952-56f02961c069","Type":"ContainerDied","Data":"c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e"}
Oct 31 19:05:30.228757 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.228663    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"
Oct 31 19:05:30.228757 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.228700    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b" event={"ID":"9fe44e72-b397-4171-9952-56f02961c069","Type":"ContainerDied","Data":"1471322de4014f664dd1e87364f450e83577ed05a3c56b70554e23e1b63fac7d"}
Oct 31 19:05:30.238468 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.238453    4484 scope.go:122] "RemoveContainer" containerID="c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680"
Oct 31 19:05:30.238786 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:05:30.238768    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680\": container with ID starting with c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680 not found: ID does not exist" containerID="c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680"
Oct 31 19:05:30.238867 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.238797    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680"} err="failed to get container status \"c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680\": rpc error: code = NotFound desc = could not find container \"c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680\": container with ID starting with c6a4b734f4e1e2c47c05e2017579f144a94721bd46ff206d2f4189872456a680 not found: ID does not exist"
Oct 31 19:05:30.238867 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.238819    4484 scope.go:122] "RemoveContainer" containerID="c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e"
Oct 31 19:05:30.248024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.248004    4484 scope.go:122] "RemoveContainer" containerID="c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e"
Oct 31 19:05:30.248299 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:05:30.248282    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e\": container with ID starting with c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e not found: ID does not exist" containerID="c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e"
Oct 31 19:05:30.248356 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.248308    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e"} err="failed to get container status \"c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e\": rpc error: code = NotFound desc = could not find container \"c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e\": container with ID starting with c24e9cf36c1f89814b9a5feb39d2a5d98fe872b599b1dc831376844ccf68aa0e not found: ID does not exist"
Oct 31 19:05:30.392499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.392452    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"]
Oct 31 19:05:30.395447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.395419    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1ef01f-6789797699-2xqs9"]
Oct 31 19:05:30.505681 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.505643    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"]
Oct 31 19:05:30.508669 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.508645    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1ef01f-7647b6dd7f-p557b"]
Oct 31 19:05:30.585975 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:30.585939    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:31.994313 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:31.994281    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8056bf6d-b304-4656-af4c-8a5b8708845b" path="/var/lib/kubelet/pods/8056bf6d-b304-4656-af4c-8a5b8708845b/volumes"
Oct 31 19:05:31.994762 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:31.994747    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe44e72-b397-4171-9952-56f02961c069" path="/var/lib/kubelet/pods/9fe44e72-b397-4171-9952-56f02961c069/volumes"
Oct 31 19:05:32.230008 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:32.229973    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:32.379079 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:32.378994    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:33.044607 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:33.044574    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"]
Oct 31 19:05:33.094836 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:33.094803    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d61cd41f-czd8l"]
Oct 31 19:05:33.993844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:33.993816    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c95c0b0-fde1-4af1-98b3-4ba00b603623" path="/var/lib/kubelet/pods/7c95c0b0-fde1-4af1-98b3-4ba00b603623/volumes"
Oct 31 19:05:35.331107 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:35.331071    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:39.632502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.632452    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"]
Oct 31 19:05:39.633043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.633030    4484 cpu_manager.go:414] "containerMap: removing container" podUID="8056bf6d-b304-4656-af4c-8a5b8708845b" containerName="payload-processing"
Oct 31 19:05:39.633088 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.633044    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8056bf6d-b304-4656-af4c-8a5b8708845b" containerName="payload-processing"
Oct 31 19:05:39.633088 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.633064    4484 cpu_manager.go:414] "containerMap: removing container" podUID="9fe44e72-b397-4171-9952-56f02961c069" containerName="payload-pre-processing"
Oct 31 19:05:39.633088 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.633069    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe44e72-b397-4171-9952-56f02961c069" containerName="payload-pre-processing"
Oct 31 19:05:39.633181 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.633174    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8056bf6d-b304-4656-af4c-8a5b8708845b" containerName="payload-processing"
Oct 31 19:05:39.633218 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.633185    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe44e72-b397-4171-9952-56f02961c069" containerName="payload-pre-processing"
Oct 31 19:05:39.637891 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.637875    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:39.695284 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.695253    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"]
Oct 31 19:05:39.706566 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.706539    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-dbceb9-serving-cert\""
Oct 31 19:05:39.789687 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.789657    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5zjc\" (UniqueName: \"kubernetes.io/projected/f4594ae3-ae84-47f7-93ca-2295d092f97c-kube-api-access-p5zjc\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:39.789872 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.789788    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:39.789922 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.789877    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4594ae3-ae84-47f7-93ca-2295d092f97c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:39.789975 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.789930    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:39.838045 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.838012    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:39.891133 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.891044    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4594ae3-ae84-47f7-93ca-2295d092f97c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:39.891133 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.891125    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:39.891363 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.891222    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zjc\" (UniqueName: \"kubernetes.io/projected/f4594ae3-ae84-47f7-93ca-2295d092f97c-kube-api-access-p5zjc\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:39.891363 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:05:39.891295    4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-serving-cert: secret "maas-api-e2e-shared-a-w2-dbceb9-serving-cert" not found
Oct 31 19:05:39.891523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.891301    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:39.891523 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:05:39.891384    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-maas-api-tls podName:f4594ae3-ae84-47f7-93ca-2295d092f97c nodeName:}" failed. No retries permitted until 2026-10-31 19:05:40.391361151 +0000 UTC m=+2686.537810998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-maas-api-tls") pod "maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28" (UID: "f4594ae3-ae84-47f7-93ca-2295d092f97c") : secret "maas-api-e2e-shared-a-w2-dbceb9-serving-cert" not found
Oct 31 19:05:39.891899 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.891874    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4594ae3-ae84-47f7-93ca-2295d092f97c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:39.894115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.894096    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:39.950065 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:39.950031    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zjc\" (UniqueName: \"kubernetes.io/projected/f4594ae3-ae84-47f7-93ca-2295d092f97c-kube-api-access-p5zjc\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:40.000633 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:40.000602    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:40.156065 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:40.155980    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:40.397093 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:40.397060    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:40.399561 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:40.399535    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:40.552136 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:40.552107    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:40.934122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:40.934028    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"]
Oct 31 19:05:40.937451 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:05:40.937423    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4594ae3_ae84_47f7_93ca_2295d092f97c.slice/crio-fc0cd3778f4f955abc92168aeaffffe22c4b2f016f960c9b649bf8dc965ed9d6 WatchSource:0}: Error finding container fc0cd3778f4f955abc92168aeaffffe22c4b2f016f960c9b649bf8dc965ed9d6: Status 404 returned error can't find the container with id fc0cd3778f4f955abc92168aeaffffe22c4b2f016f960c9b649bf8dc965ed9d6
Oct 31 19:05:40.938816 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:40.938796    4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:05:41.302855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:41.302814    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28" event={"ID":"f4594ae3-ae84-47f7-93ca-2295d092f97c","Type":"ContainerStarted","Data":"fc0cd3778f4f955abc92168aeaffffe22c4b2f016f960c9b649bf8dc965ed9d6"}
Oct 31 19:05:42.312217 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:42.312176    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28" event={"ID":"f4594ae3-ae84-47f7-93ca-2295d092f97c","Type":"ContainerStarted","Data":"04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59"}
Oct 31 19:05:42.312622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:42.312246    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:42.428863 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:42.428811    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28" podStartSLOduration=2.610961142 podStartE2EDuration="3.428797191s" totalImagesPullingTime="817.836049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:05:42.427061239 +0000 UTC m=+2688.573511104" watchObservedRunningTime="2026-10-31 19:05:42.428797191 +0000 UTC m=+2688.575247067"
Oct 31 19:05:43.937393 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:43.937359    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:05:44.696735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:44.696681    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:48.324187 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:48.324159    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:05:49.452010 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:49.451969    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:54.150074 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:54.150038    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:54.609814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:54.609788    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:05:54.611462 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:54.611442    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:05:55.426350 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:55.426258    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:55.469033 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:05:55.469000    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:05:58.806051 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:58.806015    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:05:59.866746 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:05:59.866713    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:03.403018 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:03.402979    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:08.182701 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:08.182666    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:11.926820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:11.926785    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:12.339339 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:12.339299    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:13.072764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.072723    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:13.115395 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.115366    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"]
Oct 31 19:06:13.120935 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.120912    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.197157 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.197129    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-c4731a-openshift-default-dockercfg-vpqcm\""
Oct 31 19:06:13.218952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.218919    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"]
Oct 31 19:06:13.285541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.285504    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:13.286826 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.286797    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-socket\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.286990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.286968    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-envoy\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.287124 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.287101    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-token\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.287175 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.287159    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-certs\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.287225 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.287194    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-data\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.287300 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.287281    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-podinfo\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.287396 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.287379    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istiod-ca-cert\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.287510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.287458    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-credential-socket\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.287610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.287587    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv2wq\" (UniqueName: \"kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-kube-api-access-kv2wq\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.388691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.388602    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-certs\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.388691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.388643    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-data\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.388691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.388668    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-podinfo\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.388989 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.388700    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istiod-ca-cert\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.388989 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.388774    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-credential-socket\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.388989 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.388922    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv2wq\" (UniqueName: \"kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-kube-api-access-kv2wq\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.389139 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.389011    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-socket\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.389139 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.389082    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-certs\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.389139 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.389118    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-envoy\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.389139 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.389124    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-data\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.389331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.389226    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-token\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.389550 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.389530    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-credential-socket\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.389550 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.389540    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istiod-ca-cert\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.389809 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.389787    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-socket\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.391348 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.391327    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-podinfo\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.391444 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.391426    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-envoy\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.446770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.446742    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-token\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.446956 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.446858    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv2wq\" (UniqueName: \"kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-kube-api-access-kv2wq\") pod \"e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") " pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:13.474876 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.474801    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:13.629310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.629258    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:13.736941 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:13.736915    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:14.166260 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:14.166221    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"]
Oct 31 19:06:14.167218 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:06:14.167194    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb435e8f9_377a_4d33_bee0_a5e03e4eb4d7.slice/crio-c16437a194ff89a558b4e8469cc27dc43490b555a78bee44b2797abd83e40724 WatchSource:0}: Error finding container c16437a194ff89a558b4e8469cc27dc43490b555a78bee44b2797abd83e40724: Status 404 returned error can't find the container with id c16437a194ff89a558b4e8469cc27dc43490b555a78bee44b2797abd83e40724
Oct 31 19:06:14.169780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:14.169740    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:06:14.169902 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:14.169820    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:06:14.169902 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:14.169862    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:06:14.512565 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:14.512529    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" event={"ID":"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7","Type":"ContainerStarted","Data":"8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2"}
Oct 31 19:06:14.512751 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:14.512585    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" event={"ID":"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7","Type":"ContainerStarted","Data":"c16437a194ff89a558b4e8469cc27dc43490b555a78bee44b2797abd83e40724"}
Oct 31 19:06:14.633584 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:14.633534    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" podStartSLOduration=1.6335193590000001 podStartE2EDuration="1.633519359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:06:14.630312042 +0000 UTC m=+2720.776761923" watchObservedRunningTime="2026-10-31 19:06:14.633519359 +0000 UTC m=+2720.779969223"
Oct 31 19:06:14.738020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:14.737990    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:15.743433 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:15.743398    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:16.528956 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:16.528923    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:16.530005 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:16.529986    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:06:17.469545 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:17.469509    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:17.685919 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:17.685879    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:17.843191 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:17.843151    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:19.415066 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:19.415026    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:06:19.655892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:19.655858    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:21.516730 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:21.516701    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:21.719918 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:21.719882    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:24.528992 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:24.528954    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:24.736714 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:24.736679    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:24.896668 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:24.896578    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:25.468975 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:06:25.468941    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:06:25.477549 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:25.477520    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:25.941012 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:25.940973    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:26.098098 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:26.097978    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:26.415276 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:26.415243    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:26.633844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:26.633810    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:26.797280 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:26.797242    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:26.943107 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:26.943070    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:29.906616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:29.906576    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:31.141919 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:31.141879    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:31.367260 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:31.367222    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:34.388032 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.388000    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:34.558582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.558539    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:34.764203 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.764162    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"]
Oct 31 19:06:34.768697 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.768681    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"
Oct 31 19:06:34.799381 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.799343    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5e6ede-697c-47c4-b53b-99a417db3f2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg\" (UID: \"0e5e6ede-697c-47c4-b53b-99a417db3f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"
Oct 31 19:06:34.799535 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.799507    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs7ld\" (UniqueName: \"kubernetes.io/projected/0e5e6ede-697c-47c4-b53b-99a417db3f2d-kube-api-access-vs7ld\") pod \"payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg\" (UID: \"0e5e6ede-697c-47c4-b53b-99a417db3f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"
Oct 31 19:06:34.821264 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.821238    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d22e36\""
Oct 31 19:06:34.874852 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.874819    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"]
Oct 31 19:06:34.893953 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.893918    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"]
Oct 31 19:06:34.898519 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.898491    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"
Oct 31 19:06:34.900379 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.900336    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66\" (UID: \"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"
Oct 31 19:06:34.900684 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.900547    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5e6ede-697c-47c4-b53b-99a417db3f2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg\" (UID: \"0e5e6ede-697c-47c4-b53b-99a417db3f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"
Oct 31 19:06:34.900684 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.900624    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sc4l\" (UniqueName: \"kubernetes.io/projected/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-kube-api-access-9sc4l\") pod \"payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66\" (UID: \"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"
Oct 31 19:06:34.900912 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.900730    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7ld\" (UniqueName: \"kubernetes.io/projected/0e5e6ede-697c-47c4-b53b-99a417db3f2d-kube-api-access-vs7ld\") pod \"payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg\" (UID: \"0e5e6ede-697c-47c4-b53b-99a417db3f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"
Oct 31 19:06:34.903541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.901908    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5e6ede-697c-47c4-b53b-99a417db3f2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg\" (UID: \"0e5e6ede-697c-47c4-b53b-99a417db3f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"
Oct 31 19:06:34.959229 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.959195    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d22e36-dockercfg-bslvk\""
Oct 31 19:06:34.962271 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.962245    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7ld\" (UniqueName: \"kubernetes.io/projected/0e5e6ede-697c-47c4-b53b-99a417db3f2d-kube-api-access-vs7ld\") pod \"payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg\" (UID: \"0e5e6ede-697c-47c4-b53b-99a417db3f2d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"
Oct 31 19:06:34.962899 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:34.962869    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"]
Oct 31 19:06:35.002215 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.002185    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66\" (UID: \"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"
Oct 31 19:06:35.002413 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.002325    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sc4l\" (UniqueName: \"kubernetes.io/projected/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-kube-api-access-9sc4l\") pod \"payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66\" (UID: \"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"
Oct 31 19:06:35.003132 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.003106    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66\" (UID: \"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"
Oct 31 19:06:35.059913 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.059846    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sc4l\" (UniqueName: \"kubernetes.io/projected/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-kube-api-access-9sc4l\") pod \"payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66\" (UID: \"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"
Oct 31 19:06:35.070677 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.070645    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:35.083296 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.083275    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"
Oct 31 19:06:35.225999 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.225962    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"
Oct 31 19:06:35.468197 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.468168    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"]
Oct 31 19:06:35.598965 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.598938    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"]
Oct 31 19:06:35.599573 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:06:35.599548    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff4f7c3_15e2_4cf5_8c7b_d02ce9bbdf3a.slice/crio-22c7d0dc6437dfc0755267cd7d5238d65ee5d15aa2b456091e9a87d436de1f95 WatchSource:0}: Error finding container 22c7d0dc6437dfc0755267cd7d5238d65ee5d15aa2b456091e9a87d436de1f95: Status 404 returned error can't find the container with id 22c7d0dc6437dfc0755267cd7d5238d65ee5d15aa2b456091e9a87d436de1f95
Oct 31 19:06:35.672425 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.672377    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66" event={"ID":"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a","Type":"ContainerStarted","Data":"a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522"}
Oct 31 19:06:35.672586 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.672460    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66" event={"ID":"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a","Type":"ContainerStarted","Data":"22c7d0dc6437dfc0755267cd7d5238d65ee5d15aa2b456091e9a87d436de1f95"}
Oct 31 19:06:35.672665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.672632    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"
Oct 31 19:06:35.674093 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.674071    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg" event={"ID":"0e5e6ede-697c-47c4-b53b-99a417db3f2d","Type":"ContainerStarted","Data":"774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab"}
Oct 31 19:06:35.674213 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.674115    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"
Oct 31 19:06:35.674213 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.674131    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg" event={"ID":"0e5e6ede-697c-47c4-b53b-99a417db3f2d","Type":"ContainerStarted","Data":"c0a3fa4f78125151723824da8ed774556b2581caa4c00234621b1e64e3a21166"}
Oct 31 19:06:35.787894 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.787839    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66" podStartSLOduration=1.7878254660000001 podStartE2EDuration="1.787825466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:06:35.785747718 +0000 UTC m=+2741.932197584" watchObservedRunningTime="2026-10-31 19:06:35.787825466 +0000 UTC m=+2741.934275353"
Oct 31 19:06:35.902623 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:35.902572    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg" podStartSLOduration=1.902555103 podStartE2EDuration="1.902555103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:06:35.898865107 +0000 UTC m=+2742.045314972" watchObservedRunningTime="2026-10-31 19:06:35.902555103 +0000 UTC m=+2742.049004965"
Oct 31 19:06:40.150538 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:40.150508    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:43.156776 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:43.156744    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:43.651365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:43.651324    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:44.295938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:44.295655    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:44.543664 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:44.543620    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:45.267408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:45.267378    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:46.685651 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:46.685618    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"
Oct 31 19:06:46.686182 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:46.685883    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"
Oct 31 19:06:47.174658 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:47.174613    4484 patch_prober.go:29] interesting pod/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:06:47.174871 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:47.174691    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:06:48.766685 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:48.766652    4484 generic.go:356] "Generic (PLEG): container finished" podID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerID="4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7" exitCode=0
Oct 31 19:06:48.767139 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:48.766731    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" event={"ID":"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d","Type":"ContainerDied","Data":"4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7"}
Oct 31 19:06:48.767187 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:48.767173    4484 scope.go:122] "RemoveContainer" containerID="4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7"
Oct 31 19:06:48.768718 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:48.768691    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:06:48.768790 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:48.768761    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:06:48.768827 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:48.768793    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:06:49.578516 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:49.574109    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:49.748605 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:49.748573    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:49.791971 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:49.791943    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" event={"ID":"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d","Type":"ContainerStarted","Data":"a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585"}
Oct 31 19:06:49.911508 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:49.911376    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:50.016466 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:50.016437    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:06:50.016736 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:50.016716    4484 patch_prober.go:29] interesting pod/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 31 19:06:50.016820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:50.016768    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 31 19:06:50.212920 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:50.212832    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:51.016166 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:51.016133    4484 patch_prober.go:29] interesting pod/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 31 19:06:51.016616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:51.016184    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 31 19:06:52.016456 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:52.016417    4484 patch_prober.go:29] interesting pod/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 31 19:06:52.016905 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:52.016506    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 31 19:06:52.886848 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:52.886792    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:53.016534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.016461    4484 patch_prober.go:29] interesting pod/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 31 19:06:53.017127 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.017086    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 31 19:06:53.090205 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.090166    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:53.245454 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.245421    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"]
Oct 31 19:06:53.249693 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.249671    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"
Oct 31 19:06:53.304858 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.304823    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"]
Oct 31 19:06:53.310770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.310738    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc\" (UID: \"b0cc9f9a-c164-4c68-a62c-9b8832aa0850\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"
Oct 31 19:06:53.311156 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.311133    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qt8\" (UniqueName: \"kubernetes.io/projected/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-kube-api-access-97qt8\") pod \"payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc\" (UID: \"b0cc9f9a-c164-4c68-a62c-9b8832aa0850\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"
Oct 31 19:06:53.352503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.351716    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6cfd68-dockercfg-8fksw\""
Oct 31 19:06:53.352503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.351832    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6cfd68\""
Oct 31 19:06:53.414641 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.414602    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc\" (UID: \"b0cc9f9a-c164-4c68-a62c-9b8832aa0850\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"
Oct 31 19:06:53.414846 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.414826    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qt8\" (UniqueName: \"kubernetes.io/projected/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-kube-api-access-97qt8\") pod \"payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc\" (UID: \"b0cc9f9a-c164-4c68-a62c-9b8832aa0850\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"
Oct 31 19:06:53.416101 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.416070    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc\" (UID: \"b0cc9f9a-c164-4c68-a62c-9b8832aa0850\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"
Oct 31 19:06:53.491500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.490274    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qt8\" (UniqueName: \"kubernetes.io/projected/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-kube-api-access-97qt8\") pod \"payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc\" (UID: \"b0cc9f9a-c164-4c68-a62c-9b8832aa0850\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"
Oct 31 19:06:53.584708 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.584620    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"
Oct 31 19:06:53.959879 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:06:53.959843    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cc9f9a_c164_4c68_a62c_9b8832aa0850.slice/crio-ed6a6ebdbed8feaf1361e677c4c895875a04c1e89ec05cf794b2b68757544b52 WatchSource:0}: Error finding container ed6a6ebdbed8feaf1361e677c4c895875a04c1e89ec05cf794b2b68757544b52: Status 404 returned error can't find the container with id ed6a6ebdbed8feaf1361e677c4c895875a04c1e89ec05cf794b2b68757544b52
Oct 31 19:06:53.960515 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:53.960402    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"]
Oct 31 19:06:54.017105 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:54.017079    4484 patch_prober.go:29] interesting pod/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 31 19:06:54.017467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:54.017150    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 31 19:06:54.826757 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:54.826718    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc" event={"ID":"b0cc9f9a-c164-4c68-a62c-9b8832aa0850","Type":"ContainerStarted","Data":"b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d"}
Oct 31 19:06:54.826977 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:54.826789    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"
Oct 31 19:06:54.826977 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:54.826813    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc" event={"ID":"b0cc9f9a-c164-4c68-a62c-9b8832aa0850","Type":"ContainerStarted","Data":"ed6a6ebdbed8feaf1361e677c4c895875a04c1e89ec05cf794b2b68757544b52"}
Oct 31 19:06:54.942519 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:54.942452    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc" podStartSLOduration=1.942433187 podStartE2EDuration="1.942433187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:06:54.940181754 +0000 UTC m=+2761.086631618" watchObservedRunningTime="2026-10-31 19:06:54.942433187 +0000 UTC m=+2761.088883050"
Oct 31 19:06:55.020999 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:55.020971    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:06:55.021528 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:55.021508    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:06:55.022083 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:55.022067    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:06:55.077012 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:55.076915    4484 scope.go:122] "RemoveContainer" containerID="39eb70184ec1c75e84be7aa4d14efb26db99128faaee7bf286fc7d97b244788e"
Oct 31 19:06:55.286845 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:55.286714    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:55.470259 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:06:55.470170    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:06:55.752604 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:55.752571    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:06:59.946163 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:06:59.946125    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:00.198650 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:00.198563    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:02.251836 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:02.251801    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:07:04.355684 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:04.355644    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:04.659196 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:04.659106    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:05.081150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:05.081114    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:05.355358 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:05.355259    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:05.836514 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:05.836463    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"
Oct 31 19:07:09.527389 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:09.527339    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:09.690357 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:09.690297    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:09.847892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:09.847799    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:12.778197 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:12.778162    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:12.975618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:12.975586    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:13.172761 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.172687    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"]
Oct 31 19:07:13.178580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.178556    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"
Oct 31 19:07:13.234646 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.234619    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"]
Oct 31 19:07:13.263368 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.263318    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc\" (UID: \"7f6a2a64-1451-4acc-8caf-17f9b16c30e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"
Oct 31 19:07:13.264612 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.264583    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xw2x\" (UniqueName: \"kubernetes.io/projected/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-kube-api-access-6xw2x\") pod \"payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc\" (UID: \"7f6a2a64-1451-4acc-8caf-17f9b16c30e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"
Oct 31 19:07:13.289636 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.289568    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-d1ff32\""
Oct 31 19:07:13.291875 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.291816    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-d1ff32-dockercfg-5pw72\""
Oct 31 19:07:13.366262 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.366219    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc\" (UID: \"7f6a2a64-1451-4acc-8caf-17f9b16c30e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"
Oct 31 19:07:13.366508 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.366302    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xw2x\" (UniqueName: \"kubernetes.io/projected/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-kube-api-access-6xw2x\") pod \"payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc\" (UID: \"7f6a2a64-1451-4acc-8caf-17f9b16c30e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"
Oct 31 19:07:13.367019 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.366994    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc\" (UID: \"7f6a2a64-1451-4acc-8caf-17f9b16c30e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"
Oct 31 19:07:13.423971 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.423896    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xw2x\" (UniqueName: \"kubernetes.io/projected/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-kube-api-access-6xw2x\") pod \"payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc\" (UID: \"7f6a2a64-1451-4acc-8caf-17f9b16c30e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"
Oct 31 19:07:13.492234 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.492201    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"
Oct 31 19:07:13.871901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.871651    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"]
Oct 31 19:07:13.957956 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.957927    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc" event={"ID":"7f6a2a64-1451-4acc-8caf-17f9b16c30e8","Type":"ContainerStarted","Data":"cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444"}
Oct 31 19:07:13.958074 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.957975    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc" event={"ID":"7f6a2a64-1451-4acc-8caf-17f9b16c30e8","Type":"ContainerStarted","Data":"27acfc2093b903ff6e4c8aee6560150034626c4fcde69e9aa6c6f2c99a658e9a"}
Oct 31 19:07:13.958074 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:13.958021    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"
Oct 31 19:07:14.074557 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:14.074499    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc" podStartSLOduration=1.074456803 podStartE2EDuration="1.074456803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:07:14.07121368 +0000 UTC m=+2780.217663544" watchObservedRunningTime="2026-10-31 19:07:14.074456803 +0000 UTC m=+2780.220906668"
Oct 31 19:07:14.685518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:14.685467    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:14.855741 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:14.855703    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:15.010297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:15.010264    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:15.636004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:15.635968    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:19.474974 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:19.474939    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:20.044221 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:20.044178    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:24.761235 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:24.761192    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:24.969557 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:24.969527    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"
Oct 31 19:07:25.468892 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:07:25.468858    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:07:29.186454 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.186416    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"]
Oct 31 19:07:29.191233 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.191207    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.247298 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.247268    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"]
Oct 31 19:07:29.266981 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.266948    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d25c45a-e75d-4299-a206-c3875e851f43-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.267153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.267067    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlwhf\" (UniqueName: \"kubernetes.io/projected/7d25c45a-e75d-4299-a206-c3875e851f43-kube-api-access-dlwhf\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.267153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.267107    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.267290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.267277    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.293788 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.293759    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d22e36-serving-cert\""
Oct 31 19:07:29.368562 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.368524    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.368780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.368599    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d25c45a-e75d-4299-a206-c3875e851f43-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.368780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.368661    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlwhf\" (UniqueName: \"kubernetes.io/projected/7d25c45a-e75d-4299-a206-c3875e851f43-kube-api-access-dlwhf\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.368780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.368682    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.368780 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:07:29.368706    4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-serving-cert: secret "maas-api-e2e-shared-b-w3-d22e36-serving-cert" not found
Oct 31 19:07:29.369033 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:07:29.368816    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-maas-api-tls podName:7d25c45a-e75d-4299-a206-c3875e851f43 nodeName:}" failed. No retries permitted until 2026-10-31 19:07:29.86879388 +0000 UTC m=+2796.015243743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-maas-api-tls") pod "maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p" (UID: "7d25c45a-e75d-4299-a206-c3875e851f43") : secret "maas-api-e2e-shared-b-w3-d22e36-serving-cert" not found
Oct 31 19:07:29.369361 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.369342    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d25c45a-e75d-4299-a206-c3875e851f43-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.371244 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.371217    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.399242 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.399212    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:29.427092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.427050    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlwhf\" (UniqueName: \"kubernetes.io/projected/7d25c45a-e75d-4299-a206-c3875e851f43-kube-api-access-dlwhf\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.610426 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.610387    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:29.876384 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.876286    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:29.878985 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:29.878954    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:30.105398 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:30.105364    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:30.483002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:30.482973    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"]
Oct 31 19:07:30.483575 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:07:30.483544    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d25c45a_e75d_4299_a206_c3875e851f43.slice/crio-66f2f2f71a6dcfc72150464e35cae22f53be9f54aa1ed3740b59d332f14cdb6c WatchSource:0}: Error finding container 66f2f2f71a6dcfc72150464e35cae22f53be9f54aa1ed3740b59d332f14cdb6c: Status 404 returned error can't find the container with id 66f2f2f71a6dcfc72150464e35cae22f53be9f54aa1ed3740b59d332f14cdb6c
Oct 31 19:07:31.071538 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:31.071502    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p" event={"ID":"7d25c45a-e75d-4299-a206-c3875e851f43","Type":"ContainerStarted","Data":"66f2f2f71a6dcfc72150464e35cae22f53be9f54aa1ed3740b59d332f14cdb6c"}
Oct 31 19:07:32.079937 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:32.079906    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p" event={"ID":"7d25c45a-e75d-4299-a206-c3875e851f43","Type":"ContainerStarted","Data":"a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e"}
Oct 31 19:07:32.080341 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:32.080037    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:32.197908 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:32.197860    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p" podStartSLOduration=2.406117397 podStartE2EDuration="3.197844456s" totalImagesPullingTime="791.727059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:07:32.19434691 +0000 UTC m=+2798.340796768" watchObservedRunningTime="2026-10-31 19:07:32.197844456 +0000 UTC m=+2798.344294320"
Oct 31 19:07:34.190652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:34.190617    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:35.860531 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:35.860497    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:07:38.092001 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:38.091973    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:07:38.928338 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:38.928295    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:43.570861 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:43.570824    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:46.173388 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:46.173359    4484 generic.go:356] "Generic (PLEG): container finished" podID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerID="8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2" exitCode=0
Oct 31 19:07:46.173860 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:46.173434    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" event={"ID":"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7","Type":"ContainerDied","Data":"8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2"}
Oct 31 19:07:46.173916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:46.173854    4484 scope.go:122] "RemoveContainer" containerID="8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2"
Oct 31 19:07:46.175394 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:46.175363    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:07:46.175466 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:46.175437    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:07:46.175544 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:46.175485    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:07:46.529453 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:46.529391    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:07:47.193362 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:47.193324    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" event={"ID":"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7","Type":"ContainerStarted","Data":"6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8"}
Oct 31 19:07:47.737954 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:47.737925    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:07:47.739074 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:47.739052    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 19:07:47.739170 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:47.739107    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 19:07:48.360606 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:48.360572    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:48.737195 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:48.737170    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 19:07:48.737360 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:48.737231    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 19:07:49.737949 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:49.737915    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 19:07:49.738342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:49.737969    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 19:07:50.737680 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:50.737646    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 19:07:50.737858 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:50.737710    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 19:07:51.737762 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:51.737726    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 19:07:51.738238 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:51.737798    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 19:07:52.742009 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:52.741982    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:07:52.742412 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:52.742250    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:07:52.743016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:52.742996    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:07:52.856074 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:52.856045    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"]
Oct 31 19:07:52.860837 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:52.860813    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:52.876351 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:52.876319    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"]
Oct 31 19:07:52.921230 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:52.921198    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6cfd68-serving-cert\""
Oct 31 19:07:53.024447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.024351    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ff7b73-ecc2-4890-9ae0-284e55764162-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.024654 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.024538    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.024654 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.024646    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.024793 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.024650    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:53.024793 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.024715    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snqq4\" (UniqueName: \"kubernetes.io/projected/41ff7b73-ecc2-4890-9ae0-284e55764162-kube-api-access-snqq4\") pod \"maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.126152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.126124    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.126339 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.126229    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.126339 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.126267    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snqq4\" (UniqueName: \"kubernetes.io/projected/41ff7b73-ecc2-4890-9ae0-284e55764162-kube-api-access-snqq4\") pod \"maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.126467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.126375    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ff7b73-ecc2-4890-9ae0-284e55764162-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.127153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.127127    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ff7b73-ecc2-4890-9ae0-284e55764162-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.128811 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.128788    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.128892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.128878    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.183506 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.183458    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snqq4\" (UniqueName: \"kubernetes.io/projected/41ff7b73-ecc2-4890-9ae0-284e55764162-kube-api-access-snqq4\") pod \"maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.188960 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.188929    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:53.333094 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.333007    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:53.478155 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.478118    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:53.854567 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:53.854529    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"]
Oct 31 19:07:53.854900 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:07:53.854527    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ff7b73_ecc2_4890_9ae0_284e55764162.slice/crio-3807ab3a2a283b472df923099d9e9c2d26525897f99d60b6bc52151364fc06c7 WatchSource:0}: Error finding container 3807ab3a2a283b472df923099d9e9c2d26525897f99d60b6bc52151364fc06c7: Status 404 returned error can't find the container with id 3807ab3a2a283b472df923099d9e9c2d26525897f99d60b6bc52151364fc06c7
Oct 31 19:07:54.253864 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:54.253837    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw" event={"ID":"41ff7b73-ecc2-4890-9ae0-284e55764162","Type":"ContainerStarted","Data":"3807ab3a2a283b472df923099d9e9c2d26525897f99d60b6bc52151364fc06c7"}
Oct 31 19:07:55.262393 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:55.262348    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw" event={"ID":"41ff7b73-ecc2-4890-9ae0-284e55764162","Type":"ContainerStarted","Data":"772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5"}
Oct 31 19:07:55.262830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:55.262419    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:07:55.380735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:55.380678    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw" podStartSLOduration=2.600493197 podStartE2EDuration="3.380661248s" totalImagesPullingTime="780.168051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:07:55.378212981 +0000 UTC m=+2821.524662845" watchObservedRunningTime="2026-10-31 19:07:55.380661248 +0000 UTC m=+2821.527111113"
Oct 31 19:07:55.469152 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:07:55.469119    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:07:56.163213 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:56.163175    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:07:57.960941 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:07:57.960908    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:00.522298 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:00.522254    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:01.273575 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:01.273543    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:08:02.661387 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:02.661349    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:07.261122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:07.261034    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:10.200976 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:10.200941    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:08:11.896399 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:11.896366    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:16.430554 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:16.430519    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:16.609600 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:16.609563    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:16.754821 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:16.754792    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:21.488939 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:21.488901    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:25.472070 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:08:25.472024    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:08:26.149941 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:26.149888    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:30.855771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:30.855730    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:34.047223 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:34.047187    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"]
Oct 31 19:08:34.097634 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:34.097603    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f69e81d-vg59l"]
Oct 31 19:08:35.848358 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:35.848319    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:35.995259 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:35.995227    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c78898-1640-4522-8ebd-5d8fe9a0a49a" path="/var/lib/kubelet/pods/74c78898-1640-4522-8ebd-5d8fe9a0a49a/volumes"
Oct 31 19:08:36.084008 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:36.083968    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:39.163227 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:39.163188    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:40.684723 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:40.684687    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:45.454334 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:45.454290    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:49.093326 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:49.093283    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"]
Oct 31 19:08:49.096838 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:49.096805    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19bfd618-tk46n"]
Oct 31 19:08:49.846433 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:49.846390    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:49.995166 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:49.995128    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="359d0aca-1468-4717-8363-508cf99704b4" path="/var/lib/kubelet/pods/359d0aca-1468-4717-8363-508cf99704b4/volumes"
Oct 31 19:08:50.350393 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:50.350360    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:54.993001 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:54.992965    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:55.471118 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:08:55.471034    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:08:56.451054 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:56.451009    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:08:59.666086 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:08:59.666048    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:00.859983 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:00.859934    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:04.326215 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:04.326182    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:09:04.330102 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:04.330068    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:09.046122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:09.046077    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:10.057886 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:10.057845    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"]
Oct 31 19:09:10.108554 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:10.108519    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54f52fe0-r82g7"]
Oct 31 19:09:11.995308 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:11.995272    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af5672f-e6a1-4322-84d3-3577ac3fc55b" path="/var/lib/kubelet/pods/5af5672f-e6a1-4322-84d3-3577ac3fc55b/volumes"
Oct 31 19:09:12.902895 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:12.902849    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:14.066048 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:14.065977    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:18.767680 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:18.767630    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:23.045055 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:23.045011    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"]
Oct 31 19:09:23.095491 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:23.095430    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02c3bcac-v7wk6"]
Oct 31 19:09:23.457179 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:23.457084    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:23.995527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:23.995492    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83" path="/var/lib/kubelet/pods/5f2e6eff-8e10-4f9a-94b0-baeb60a9eb83/volumes"
Oct 31 19:09:25.471934 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:09:25.471897    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:09:28.163928 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:28.163887    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:32.862817 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:32.862770    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:35.537017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:35.536979    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:09:37.853011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:37.852973    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:38.009228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:38.009193    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:38.301444 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:38.301396    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:38.923797 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:38.923756    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:39.260523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:39.260456    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:40.593355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:40.593319    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:42.656263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:42.656224    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:43.046553 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:43.046515    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:43.216808 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:43.216774    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:43.379248 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:43.379157    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:45.333176 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:45.333139    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:45.416164 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:45.416122    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"]
Oct 31 19:09:45.416724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:45.416686    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6" podUID="f8dd4393-2398-439f-aeac-8cb8abe74d77" containerName="istio-proxy" containerID="cri-o://adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce" gracePeriod=30
Oct 31 19:09:45.839437 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:45.839395    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:46.315313 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:46.315276    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:46.467069 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:46.467030    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:46.616380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:46.616223    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:47.848579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:47.848537    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:50.697505 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.697406    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:09:50.809573 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.809539    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-certs\") pod \"f8dd4393-2398-439f-aeac-8cb8abe74d77\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") "
Oct 31 19:09:50.809770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.809617    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-kube-api-access-w86pr\" (UniqueName: \"kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-kube-api-access-w86pr\") pod \"f8dd4393-2398-439f-aeac-8cb8abe74d77\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") "
Oct 31 19:09:50.809770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.809707    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8dd4393-2398-439f-aeac-8cb8abe74d77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dd4393-2398-439f-aeac-8cb8abe74d77-istiod-ca-cert\") pod \"f8dd4393-2398-439f-aeac-8cb8abe74d77\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") "
Oct 31 19:09:50.809770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.809765    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-podinfo\") pod \"f8dd4393-2398-439f-aeac-8cb8abe74d77\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") "
Oct 31 19:09:50.809894 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.809816    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-certs" pod "f8dd4393-2398-439f-aeac-8cb8abe74d77" (UID: "f8dd4393-2398-439f-aeac-8cb8abe74d77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:50.809894 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.809888    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-data\") pod \"f8dd4393-2398-439f-aeac-8cb8abe74d77\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") "
Oct 31 19:09:50.809998 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.809922    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-credential-socket\") pod \"f8dd4393-2398-439f-aeac-8cb8abe74d77\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") "
Oct 31 19:09:50.809998 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.809964    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-envoy\") pod \"f8dd4393-2398-439f-aeac-8cb8abe74d77\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") "
Oct 31 19:09:50.810110 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.810020    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-socket\") pod \"f8dd4393-2398-439f-aeac-8cb8abe74d77\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") "
Oct 31 19:09:50.810110 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.810080    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-data" pod "f8dd4393-2398-439f-aeac-8cb8abe74d77" (UID: "f8dd4393-2398-439f-aeac-8cb8abe74d77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:50.810110 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.810083    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-token\" (UniqueName: \"kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-token\") pod \"f8dd4393-2398-439f-aeac-8cb8abe74d77\" (UID: \"f8dd4393-2398-439f-aeac-8cb8abe74d77\") "
Oct 31 19:09:50.810334 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.810184    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8dd4393-2398-439f-aeac-8cb8abe74d77-istiod-ca-cert" pod "f8dd4393-2398-439f-aeac-8cb8abe74d77" (UID: "f8dd4393-2398-439f-aeac-8cb8abe74d77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:09:50.810334 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.810249    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-credential-socket" pod "f8dd4393-2398-439f-aeac-8cb8abe74d77" (UID: "f8dd4393-2398-439f-aeac-8cb8abe74d77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:50.810431 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.810380    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-socket" pod "f8dd4393-2398-439f-aeac-8cb8abe74d77" (UID: "f8dd4393-2398-439f-aeac-8cb8abe74d77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:50.810868 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.810675    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:50.810868 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.810697    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:50.810868 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.810712    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8dd4393-2398-439f-aeac-8cb8abe74d77-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:50.810868 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.810723    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:50.810868 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.810733    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:50.812239 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.812215    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-podinfo" pod "f8dd4393-2398-439f-aeac-8cb8abe74d77" (UID: "f8dd4393-2398-439f-aeac-8cb8abe74d77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:09:50.812462 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.812443    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-token" pod "f8dd4393-2398-439f-aeac-8cb8abe74d77" (UID: "f8dd4393-2398-439f-aeac-8cb8abe74d77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:09:50.812614 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.812596    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-kube-api-access-w86pr" pod "f8dd4393-2398-439f-aeac-8cb8abe74d77" (UID: "f8dd4393-2398-439f-aeac-8cb8abe74d77"). InnerVolumeSpecName "kube-api-access-w86pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:09:50.812686 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.812626    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-envoy" pod "f8dd4393-2398-439f-aeac-8cb8abe74d77" (UID: "f8dd4393-2398-439f-aeac-8cb8abe74d77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:09:50.912363 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.912306    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:50.912363 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.912353    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:50.912363 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.912370    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w86pr\" (UniqueName: \"kubernetes.io/projected/f8dd4393-2398-439f-aeac-8cb8abe74d77-kube-api-access-w86pr\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:50.912760 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:50.912384    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8dd4393-2398-439f-aeac-8cb8abe74d77-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:09:51.099174 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:51.099136    4484 generic.go:356] "Generic (PLEG): container finished" podID="f8dd4393-2398-439f-aeac-8cb8abe74d77" containerID="adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce" exitCode=0
Oct 31 19:09:51.099360 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:51.099193    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"
Oct 31 19:09:51.099360 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:51.099221    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6" event={"ID":"f8dd4393-2398-439f-aeac-8cb8abe74d77","Type":"ContainerDied","Data":"adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce"}
Oct 31 19:09:51.099360 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:51.099277    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6" event={"ID":"f8dd4393-2398-439f-aeac-8cb8abe74d77","Type":"ContainerDied","Data":"fc709dd832535d9a6b4ca862dbf34e0821fd30001f344bdb9a0e846cfb91a165"}
Oct 31 19:09:51.099360 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:51.099299    4484 scope.go:122] "RemoveContainer" containerID="adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce"
Oct 31 19:09:51.111167 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:51.111144    4484 scope.go:122] "RemoveContainer" containerID="adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce"
Oct 31 19:09:51.111539 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:09:51.111519    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce\": container with ID starting with adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce not found: ID does not exist" containerID="adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce"
Oct 31 19:09:51.111595 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:51.111551    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce"} err="failed to get container status \"adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce\": rpc error: code = NotFound desc = could not find container \"adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce\": container with ID starting with adad7912b54a31bb5279314d2223aadd2e1d16a38011bbca56f33be0b618e3ce not found: ID does not exist"
Oct 31 19:09:51.266114 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:51.266082    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"]
Oct 31 19:09:51.269228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:51.269207    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-jlfc6"]
Oct 31 19:09:51.995605 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:51.995570    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8dd4393-2398-439f-aeac-8cb8abe74d77" path="/var/lib/kubelet/pods/f8dd4393-2398-439f-aeac-8cb8abe74d77/volumes"
Oct 31 19:09:52.460851 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:52.460764    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:55.145333 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:55.145300    4484 scope.go:122] "RemoveContainer" containerID="eeb953e872ea1ce90d71557a26814e4d0321cb4e34bf65be5e3e803e1dee1fc3"
Oct 31 19:09:55.160050 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:55.160018    4484 scope.go:122] "RemoveContainer" containerID="69d3a4f95b0e53c89285e45b3186a4c9406f174432edb3936c8ad4095a64dbf3"
Oct 31 19:09:55.472274 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:09:55.472182    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:09:56.792345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:56.792304    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:57.030606 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:57.030574    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:09:57.192923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:09:57.192822    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:01.190209 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:01.190157    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:01.802509 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:01.802445    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:06.465459 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:06.465409    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:11.066277 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:11.066209    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:15.858003 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:15.857951    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:20.463742 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:20.463687    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:21.870021 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:21.869988    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:23.069003 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:23.068971    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:23.491668 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:23.491637    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"]
Oct 31 19:10:23.875857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:23.875769    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:24.032312 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:24.032130    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:24.192781 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:24.192696    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:25.384300 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:25.384265    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:25.478913 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:10:25.478874    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:10:26.999043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:26.999006    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:10:28.491942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:28.491906    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr" podUID="62fe1994-2aa0-434b-b225-2d61b887b9ee" containerName="payload-processing" containerID="cri-o://eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030" gracePeriod=25
Oct 31 19:10:28.721671 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:28.721638    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:29.602468 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:29.602431    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:30.369715 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:30.369684    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:35.044462 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:35.044415    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:38.486450 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:38.486412    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr" podUID="62fe1994-2aa0-434b-b225-2d61b887b9ee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:10:39.759366 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:39.759330    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:42.935370 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:42.935325    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:44.455402 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:44.455363    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:48.487450 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:48.487414    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr" podUID="62fe1994-2aa0-434b-b225-2d61b887b9ee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:10:48.770198 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:48.770173    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"
Oct 31 19:10:48.850367 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:48.850333    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62fe1994-2aa0-434b-b225-2d61b887b9ee-kube-api-access-xvk7c\" (UniqueName: \"kubernetes.io/projected/62fe1994-2aa0-434b-b225-2d61b887b9ee-kube-api-access-xvk7c\") pod \"62fe1994-2aa0-434b-b225-2d61b887b9ee\" (UID: \"62fe1994-2aa0-434b-b225-2d61b887b9ee\") "
Oct 31 19:10:48.850557 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:48.850433    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62fe1994-2aa0-434b-b225-2d61b887b9ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62fe1994-2aa0-434b-b225-2d61b887b9ee-plugins-config-volume\") pod \"62fe1994-2aa0-434b-b225-2d61b887b9ee\" (UID: \"62fe1994-2aa0-434b-b225-2d61b887b9ee\") "
Oct 31 19:10:48.850884 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:48.850862    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62fe1994-2aa0-434b-b225-2d61b887b9ee-plugins-config-volume" pod "62fe1994-2aa0-434b-b225-2d61b887b9ee" (UID: "62fe1994-2aa0-434b-b225-2d61b887b9ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:10:48.850974 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:48.850959    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62fe1994-2aa0-434b-b225-2d61b887b9ee-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:48.852770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:48.852744    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fe1994-2aa0-434b-b225-2d61b887b9ee-kube-api-access-xvk7c" pod "62fe1994-2aa0-434b-b225-2d61b887b9ee" (UID: "62fe1994-2aa0-434b-b225-2d61b887b9ee"). InnerVolumeSpecName "kube-api-access-xvk7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:10:48.952883 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:48.952836    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvk7c\" (UniqueName: \"kubernetes.io/projected/62fe1994-2aa0-434b-b225-2d61b887b9ee-kube-api-access-xvk7c\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:10:49.009640 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.009526    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:49.205563 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.205521    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:49.368432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.368353    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:49.488886 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.488850    4484 generic.go:356] "Generic (PLEG): container finished" podID="62fe1994-2aa0-434b-b225-2d61b887b9ee" containerID="eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030" exitCode=0
Oct 31 19:10:49.489302 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.488899    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"
Oct 31 19:10:49.489302 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.488927    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr" event={"ID":"62fe1994-2aa0-434b-b225-2d61b887b9ee","Type":"ContainerDied","Data":"eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030"}
Oct 31 19:10:49.489302 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.488981    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr" event={"ID":"62fe1994-2aa0-434b-b225-2d61b887b9ee","Type":"ContainerDied","Data":"fb8f9bb86528ccf1675128729e5c0bafcc954b147f0b14fba824f4a95af6d9be"}
Oct 31 19:10:49.489302 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.489002    4484 scope.go:122] "RemoveContainer" containerID="eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030"
Oct 31 19:10:49.506568 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.501668    4484 scope.go:122] "RemoveContainer" containerID="eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030"
Oct 31 19:10:49.506691 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:10:49.506662    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030\": container with ID starting with eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030 not found: ID does not exist" containerID="eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030"
Oct 31 19:10:49.506757 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.506705    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030"} err="failed to get container status \"eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030\": rpc error: code = NotFound desc = could not find container \"eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030\": container with ID starting with eec626241186f8e4c6bda3b4a5632618ae82fd5bb17c1f53b636212c41f98030 not found: ID does not exist"
Oct 31 19:10:49.656421 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.656320    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"]
Oct 31 19:10:49.706659 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.706632    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a2c587-558898d98c-cbzpr"]
Oct 31 19:10:49.995143 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:49.995117    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62fe1994-2aa0-434b-b225-2d61b887b9ee" path="/var/lib/kubelet/pods/62fe1994-2aa0-434b-b225-2d61b887b9ee/volumes"
Oct 31 19:10:53.868538 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:53.868490    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:54.654947 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:54.654917    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:10:54.655515 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:54.655494    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:10:55.198887 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:55.198855    4484 scope.go:122] "RemoveContainer" containerID="0141515db88ca851f95ef477b52e8dded4a60a1dc50ee996db6f34a2d52ff7bc"
Oct 31 19:10:55.208659 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:55.208634    4484 scope.go:122] "RemoveContainer" containerID="6bd90dfaeccfdf84971c9eb52b23c0a24de60ec129d4a410e0880e186eeab4cc"
Oct 31 19:10:55.469153 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:10:55.469063    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:10:57.130983 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:57.130944    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:10:58.534728 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:10:58.534677    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:01.523356 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:01.523318    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:03.507082 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:03.507025    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:05.435574 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:05.435535    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:07.799577 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:07.799537    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:09.410031 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:09.409983    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:11:12.599429 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:12.599389    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:12.798695 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:12.798664    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:17.262982 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:17.262918    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:19.222949 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:19.222903    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:19.431123 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:19.431086    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:21.965666 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:21.965624    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:25.471319 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:11:25.471276    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:11:26.640761 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:26.640728    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:31.311901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:31.311862    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:34.491609 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:34.491571    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:35.295088 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:35.295053    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:36.998574 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:36.998537    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:37.152583 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:37.152542    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:37.596317 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:37.596277    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:38.359537 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:38.359503    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:40.612848 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:40.612812    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:11:41.887923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:41.887889    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:44.344722 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:44.344686    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:45.539858 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:45.539819    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:45.703710 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:45.703677    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:46.361519 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:46.361465    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"]
Oct 31 19:11:46.361723 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:46.361701    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" podUID="6477d04c-4bca-4354-8690-fb7e46de5ca2" containerName="istio-proxy" containerID="cri-o://2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4" gracePeriod=30
Oct 31 19:11:46.707753 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:46.707646    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:49.878560 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:49.878522    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:50.854202 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:50.854166    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:51.510416 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.510385    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 31 19:11:51.510812 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.510439    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" podUID="6477d04c-4bca-4354-8690-fb7e46de5ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 31 19:11:51.638676 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.638601    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:11:51.751447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.751414    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-data\") pod \"6477d04c-4bca-4354-8690-fb7e46de5ca2\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") "
Oct 31 19:11:51.751669 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.751564    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-envoy\") pod \"6477d04c-4bca-4354-8690-fb7e46de5ca2\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") "
Oct 31 19:11:51.751669 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.751596    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-podinfo\") pod \"6477d04c-4bca-4354-8690-fb7e46de5ca2\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") "
Oct 31 19:11:51.751669 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.751613    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-kube-api-access-24zzd\" (UniqueName: \"kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-kube-api-access-24zzd\") pod \"6477d04c-4bca-4354-8690-fb7e46de5ca2\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") "
Oct 31 19:11:51.751844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.751703    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-token\" (UniqueName: \"kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-token\") pod \"6477d04c-4bca-4354-8690-fb7e46de5ca2\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") "
Oct 31 19:11:51.751844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.751720    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6477d04c-4bca-4354-8690-fb7e46de5ca2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6477d04c-4bca-4354-8690-fb7e46de5ca2-istiod-ca-cert\") pod \"6477d04c-4bca-4354-8690-fb7e46de5ca2\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") "
Oct 31 19:11:51.751844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.751731    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-data" pod "6477d04c-4bca-4354-8690-fb7e46de5ca2" (UID: "6477d04c-4bca-4354-8690-fb7e46de5ca2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:51.751844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.751756    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-certs\") pod \"6477d04c-4bca-4354-8690-fb7e46de5ca2\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") "
Oct 31 19:11:51.751844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.751782    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-credential-socket\") pod \"6477d04c-4bca-4354-8690-fb7e46de5ca2\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") "
Oct 31 19:11:51.752106 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.751874    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-socket\") pod \"6477d04c-4bca-4354-8690-fb7e46de5ca2\" (UID: \"6477d04c-4bca-4354-8690-fb7e46de5ca2\") "
Oct 31 19:11:51.752106 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.752070    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6477d04c-4bca-4354-8690-fb7e46de5ca2-istiod-ca-cert" pod "6477d04c-4bca-4354-8690-fb7e46de5ca2" (UID: "6477d04c-4bca-4354-8690-fb7e46de5ca2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:11:51.752327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.752303    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-credential-socket" pod "6477d04c-4bca-4354-8690-fb7e46de5ca2" (UID: "6477d04c-4bca-4354-8690-fb7e46de5ca2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:51.752496 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.752346    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-certs" pod "6477d04c-4bca-4354-8690-fb7e46de5ca2" (UID: "6477d04c-4bca-4354-8690-fb7e46de5ca2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:51.752496 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.752356    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-socket" pod "6477d04c-4bca-4354-8690-fb7e46de5ca2" (UID: "6477d04c-4bca-4354-8690-fb7e46de5ca2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:51.752496 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.752445    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.752496 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.752460    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.752496 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.752490    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6477d04c-4bca-4354-8690-fb7e46de5ca2-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.752886 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.752502    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.752886 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.752512    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.754511 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.753996    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-envoy" pod "6477d04c-4bca-4354-8690-fb7e46de5ca2" (UID: "6477d04c-4bca-4354-8690-fb7e46de5ca2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:11:51.754511 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.754176    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-podinfo" pod "6477d04c-4bca-4354-8690-fb7e46de5ca2" (UID: "6477d04c-4bca-4354-8690-fb7e46de5ca2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:11:51.754511 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.754213    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-token" pod "6477d04c-4bca-4354-8690-fb7e46de5ca2" (UID: "6477d04c-4bca-4354-8690-fb7e46de5ca2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:11:51.754738 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.754715    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-kube-api-access-24zzd" pod "6477d04c-4bca-4354-8690-fb7e46de5ca2" (UID: "6477d04c-4bca-4354-8690-fb7e46de5ca2"). InnerVolumeSpecName "kube-api-access-24zzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:11:51.853993 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.853952    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.853993 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.853986    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.853993 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.853996    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24zzd\" (UniqueName: \"kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-kube-api-access-24zzd\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.854252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.854006    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6477d04c-4bca-4354-8690-fb7e46de5ca2-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:11:51.922712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.922365    4484 generic.go:356] "Generic (PLEG): container finished" podID="6477d04c-4bca-4354-8690-fb7e46de5ca2" containerID="2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4" exitCode=0
Oct 31 19:11:51.922712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.922443    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"
Oct 31 19:11:51.922712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.922454    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" event={"ID":"6477d04c-4bca-4354-8690-fb7e46de5ca2","Type":"ContainerDied","Data":"2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4"}
Oct 31 19:11:51.922712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.922543    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv" event={"ID":"6477d04c-4bca-4354-8690-fb7e46de5ca2","Type":"ContainerDied","Data":"efbc2510140b407062f1a4bc73799e94b35d53076ae2f94f46600c52291ec53f"}
Oct 31 19:11:51.922712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.922576    4484 scope.go:122] "RemoveContainer" containerID="2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4"
Oct 31 19:11:51.933733 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.933715    4484 scope.go:122] "RemoveContainer" containerID="2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4"
Oct 31 19:11:51.934038 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:11:51.934018    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4\": container with ID starting with 2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4 not found: ID does not exist" containerID="2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4"
Oct 31 19:11:51.934087 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:51.934047    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4"} err="failed to get container status \"2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4\": rpc error: code = NotFound desc = could not find container \"2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4\": container with ID starting with 2cad52f2841edc43070c3be5439d19f9e9a06ed4897de4360f95588c2d9902c4 not found: ID does not exist"
Oct 31 19:11:52.091658 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:52.091623    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"]
Oct 31 19:11:52.099408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:52.099381    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-r9fpv"]
Oct 31 19:11:52.664263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:52.664229    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:53.996909 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:53.996879    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6477d04c-4bca-4354-8690-fb7e46de5ca2" path="/var/lib/kubelet/pods/6477d04c-4bca-4354-8690-fb7e46de5ca2/volumes"
Oct 31 19:11:55.469536 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:55.469501    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:55.470923 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:11:55.470891    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:11:55.622880 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:55.622793    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"]
Oct 31 19:11:55.623067 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:55.623046    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" podUID="471772d0-1583-42bb-83c1-0858499b7210" containerName="istio-proxy" containerID="cri-o://3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb" gracePeriod=30
Oct 31 19:11:55.676355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:55.676321    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:57.030723 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:57.030681    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:57.439443 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:57.439347    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:11:57.463019 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:57.462973    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"]
Oct 31 19:11:57.463275 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:57.463250    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy" containerID="cri-o://a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585" gracePeriod=30
Oct 31 19:11:57.586121 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:11:57.586080    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:00.898395 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.898336    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:12:00.960468 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.960433    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-certs\") pod \"471772d0-1583-42bb-83c1-0858499b7210\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") "
Oct 31 19:12:00.960652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.960516    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/471772d0-1583-42bb-83c1-0858499b7210-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471772d0-1583-42bb-83c1-0858499b7210-istiod-ca-cert\") pod \"471772d0-1583-42bb-83c1-0858499b7210\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") "
Oct 31 19:12:00.960652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.960583    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-kube-api-access-5w6ff\" (UniqueName: \"kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-kube-api-access-5w6ff\") pod \"471772d0-1583-42bb-83c1-0858499b7210\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") "
Oct 31 19:12:00.960776 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.960699    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-certs" pod "471772d0-1583-42bb-83c1-0858499b7210" (UID: "471772d0-1583-42bb-83c1-0858499b7210"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:00.960833 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.960808    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-envoy\") pod \"471772d0-1583-42bb-83c1-0858499b7210\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") "
Oct 31 19:12:00.960895 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.960839    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-credential-socket\") pod \"471772d0-1583-42bb-83c1-0858499b7210\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") "
Oct 31 19:12:00.960895 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.960878    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/471772d0-1583-42bb-83c1-0858499b7210-istiod-ca-cert" pod "471772d0-1583-42bb-83c1-0858499b7210" (UID: "471772d0-1583-42bb-83c1-0858499b7210"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:12:00.960998 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.960881    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/471772d0-1583-42bb-83c1-0858499b7210-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471772d0-1583-42bb-83c1-0858499b7210-istio-podinfo\") pod \"471772d0-1583-42bb-83c1-0858499b7210\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") "
Oct 31 19:12:00.960998 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.960933    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-socket\") pod \"471772d0-1583-42bb-83c1-0858499b7210\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") "
Oct 31 19:12:00.961091 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.961010    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-data\") pod \"471772d0-1583-42bb-83c1-0858499b7210\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") "
Oct 31 19:12:00.961091 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.961049    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-istio-token\" (UniqueName: \"kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-istio-token\") pod \"471772d0-1583-42bb-83c1-0858499b7210\" (UID: \"471772d0-1583-42bb-83c1-0858499b7210\") "
Oct 31 19:12:00.961200 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.961146    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-credential-socket" pod "471772d0-1583-42bb-83c1-0858499b7210" (UID: "471772d0-1583-42bb-83c1-0858499b7210"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:00.961200 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.961180    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-socket" pod "471772d0-1583-42bb-83c1-0858499b7210" (UID: "471772d0-1583-42bb-83c1-0858499b7210"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:00.961370 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.961336    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-data" pod "471772d0-1583-42bb-83c1-0858499b7210" (UID: "471772d0-1583-42bb-83c1-0858499b7210"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:00.962011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.961933    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:00.962011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.961964    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:00.962011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.961979    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:00.962011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.961994    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/471772d0-1583-42bb-83c1-0858499b7210-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:00.962011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.962007    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:00.963044 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.963022    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-kube-api-access-5w6ff" pod "471772d0-1583-42bb-83c1-0858499b7210" (UID: "471772d0-1583-42bb-83c1-0858499b7210"). InnerVolumeSpecName "kube-api-access-5w6ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:12:00.963160 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.963039    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-envoy" pod "471772d0-1583-42bb-83c1-0858499b7210" (UID: "471772d0-1583-42bb-83c1-0858499b7210"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:00.963160 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.963149    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/471772d0-1583-42bb-83c1-0858499b7210-istio-podinfo" pod "471772d0-1583-42bb-83c1-0858499b7210" (UID: "471772d0-1583-42bb-83c1-0858499b7210"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:12:00.963494 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.963461    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-istio-token" pod "471772d0-1583-42bb-83c1-0858499b7210" (UID: "471772d0-1583-42bb-83c1-0858499b7210"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:12:00.983467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.983440    4484 generic.go:356] "Generic (PLEG): container finished" podID="471772d0-1583-42bb-83c1-0858499b7210" containerID="3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb" exitCode=0
Oct 31 19:12:00.983579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.983524    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"
Oct 31 19:12:00.983619 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.983517    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" event={"ID":"471772d0-1583-42bb-83c1-0858499b7210","Type":"ContainerDied","Data":"3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb"}
Oct 31 19:12:00.983662 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.983654    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f" event={"ID":"471772d0-1583-42bb-83c1-0858499b7210","Type":"ContainerDied","Data":"fb339a121272af0ba594d73e00c2aaf2e0f900a090dfb30acd00d5a6dbea2767"}
Oct 31 19:12:00.983695 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.983680    4484 scope.go:122] "RemoveContainer" containerID="3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb"
Oct 31 19:12:00.994245 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.994227    4484 scope.go:122] "RemoveContainer" containerID="3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb"
Oct 31 19:12:00.994544 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:12:00.994524    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb\": container with ID starting with 3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb not found: ID does not exist" containerID="3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb"
Oct 31 19:12:00.994614 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:00.994554    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb"} err="failed to get container status \"3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb\": rpc error: code = NotFound desc = could not find container \"3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb\": container with ID starting with 3eec2c9197eb7546d1f90cb9d4b701c17a72ca8185771f9757a8d34873ac72fb not found: ID does not exist"
Oct 31 19:12:01.063771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:01.063745    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w6ff\" (UniqueName: \"kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-kube-api-access-5w6ff\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:01.063771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:01.063768    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/471772d0-1583-42bb-83c1-0858499b7210-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:01.063957 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:01.063779    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/471772d0-1583-42bb-83c1-0858499b7210-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:01.063957 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:01.063788    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/471772d0-1583-42bb-83c1-0858499b7210-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:01.181120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:01.181033    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"]
Oct 31 19:12:01.183966 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:01.183941    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-c8f6f"]
Oct 31 19:12:01.868994 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:01.868947    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:01.994563 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:01.994531    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="471772d0-1583-42bb-83c1-0858499b7210" path="/var/lib/kubelet/pods/471772d0-1583-42bb-83c1-0858499b7210/volumes"
Oct 31 19:12:02.344538 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.344503    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:02.737887 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.737860    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:12:02.783873 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.783839    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-socket\") pod \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") "
Oct 31 19:12:02.784078 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.783883    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-podinfo\") pod \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") "
Oct 31 19:12:02.784078 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.783920    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-token\" (UniqueName: \"kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-token\") pod \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") "
Oct 31 19:12:02.784078 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.783949    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istiod-ca-cert\") pod \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") "
Oct 31 19:12:02.784078 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.783985    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-envoy\") pod \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") "
Oct 31 19:12:02.784297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784075    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-credential-socket\") pod \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") "
Oct 31 19:12:02.784297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784131    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-socket" pod "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" (UID: "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:02.784297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784250    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-kube-api-access-jbxvj\" (UniqueName: \"kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-kube-api-access-jbxvj\") pod \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") "
Oct 31 19:12:02.784463 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784301    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-data\") pod \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") "
Oct 31 19:12:02.784463 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784385    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-certs\") pod \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\" (UID: \"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d\") "
Oct 31 19:12:02.784463 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784413    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-credential-socket" pod "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" (UID: "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:02.784652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784495    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istiod-ca-cert" pod "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" (UID: "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:12:02.784652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784626    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-data" pod "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" (UID: "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:02.784961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784832    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-certs" pod "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" (UID: "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:02.784961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784949    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:02.784961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784964    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:02.784961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784973    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:02.784961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784982    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:02.784961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.784994    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:02.786620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.786597    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-podinfo" pod "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" (UID: "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:12:02.786765 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.786643    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-kube-api-access-jbxvj" pod "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" (UID: "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d"). InnerVolumeSpecName "kube-api-access-jbxvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:12:02.786992 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.786973    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-token" pod "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" (UID: "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:12:02.787085 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.787028    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-envoy" pod "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" (UID: "5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:02.889141 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.889051    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbxvj\" (UniqueName: \"kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-kube-api-access-jbxvj\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:02.889526 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.889504    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:02.889723 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.889709    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:02.889807 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.889796    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:02.998894 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.998861    4484 generic.go:356] "Generic (PLEG): container finished" podID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerID="a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585" exitCode=0
Oct 31 19:12:02.999432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.998942    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"
Oct 31 19:12:02.999432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.998950    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" event={"ID":"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d","Type":"ContainerDied","Data":"a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585"}
Oct 31 19:12:02.999432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.999007    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md" event={"ID":"5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d","Type":"ContainerDied","Data":"8b9fd76058ec3d08737ea930ecb88f768ad0103efaf5c387e690e8739ac24202"}
Oct 31 19:12:02.999432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:02.999030    4484 scope.go:122] "RemoveContainer" containerID="a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585"
Oct 31 19:12:03.010604 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:03.010582    4484 scope.go:122] "RemoveContainer" containerID="4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7"
Oct 31 19:12:03.021207 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:03.021185    4484 scope.go:122] "RemoveContainer" containerID="a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585"
Oct 31 19:12:03.021582 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:12:03.021557    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585\": container with ID starting with a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585 not found: ID does not exist" containerID="a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585"
Oct 31 19:12:03.021679 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:03.021592    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585"} err="failed to get container status \"a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585\": rpc error: code = NotFound desc = could not find container \"a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585\": container with ID starting with a621fc7bf057c9e5c57070fde4592613f6e24635814e1fa957f7c9e93a4c4585 not found: ID does not exist"
Oct 31 19:12:03.021679 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:03.021610    4484 scope.go:122] "RemoveContainer" containerID="4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7"
Oct 31 19:12:03.021885 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:12:03.021871    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7\": container with ID starting with 4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7 not found: ID does not exist" containerID="4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7"
Oct 31 19:12:03.021931 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:03.021889    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7"} err="failed to get container status \"4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7\": rpc error: code = NotFound desc = could not find container \"4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7\": container with ID starting with 4c6ff580d90f624c83c0c32a5f13f69e720fe45531bc3315083ac296d3039ea7 not found: ID does not exist"
Oct 31 19:12:03.166077 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:03.165989    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"]
Oct 31 19:12:03.168696 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:03.168673    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7006d3-openshift-default-d8bd9cf7c-f88md"]
Oct 31 19:12:03.996331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:03.996293    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" path="/var/lib/kubelet/pods/5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d/volumes"
Oct 31 19:12:05.988161 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:05.988124    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:06.240993 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:06.240902    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:06.615250 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:06.615173    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:06.856870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:06.856840    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:07.268512 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:07.268450    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:10.277239 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:10.277209    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:10.449273 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:10.449235    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:10.603904 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:10.603822    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:12.308611 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:12.308564    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:13.519215 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.519179    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:13.693833 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.693801    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:13.915556 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.915483    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"]
Oct 31 19:12:13.916107 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916094    4484 cpu_manager.go:414] "containerMap: removing container" podUID="6477d04c-4bca-4354-8690-fb7e46de5ca2" containerName="istio-proxy"
Oct 31 19:12:13.916152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916109    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6477d04c-4bca-4354-8690-fb7e46de5ca2" containerName="istio-proxy"
Oct 31 19:12:13.916152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916126    4484 cpu_manager.go:414] "containerMap: removing container" podUID="62fe1994-2aa0-434b-b225-2d61b887b9ee" containerName="payload-processing"
Oct 31 19:12:13.916152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916131    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62fe1994-2aa0-434b-b225-2d61b887b9ee" containerName="payload-processing"
Oct 31 19:12:13.916152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916144    4484 cpu_manager.go:414] "containerMap: removing container" podUID="471772d0-1583-42bb-83c1-0858499b7210" containerName="istio-proxy"
Oct 31 19:12:13.916152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916149    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="471772d0-1583-42bb-83c1-0858499b7210" containerName="istio-proxy"
Oct 31 19:12:13.916310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916167    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy"
Oct 31 19:12:13.916310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916174    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy"
Oct 31 19:12:13.916310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916188    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f8dd4393-2398-439f-aeac-8cb8abe74d77" containerName="istio-proxy"
Oct 31 19:12:13.916310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916193    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8dd4393-2398-439f-aeac-8cb8abe74d77" containerName="istio-proxy"
Oct 31 19:12:13.916310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916207    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy"
Oct 31 19:12:13.916310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916211    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy"
Oct 31 19:12:13.916559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916323    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6477d04c-4bca-4354-8690-fb7e46de5ca2" containerName="istio-proxy"
Oct 31 19:12:13.916559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916334    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="62fe1994-2aa0-434b-b225-2d61b887b9ee" containerName="payload-processing"
Oct 31 19:12:13.916559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916343    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="471772d0-1583-42bb-83c1-0858499b7210" containerName="istio-proxy"
Oct 31 19:12:13.916559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916352    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy"
Oct 31 19:12:13.916559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916360    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8dd4393-2398-439f-aeac-8cb8abe74d77" containerName="istio-proxy"
Oct 31 19:12:13.916559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.916369    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5113a87a-ad1d-4ba0-b056-d03fd4fd6a0d" containerName="istio-proxy"
Oct 31 19:12:13.921441 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.921425    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"
Oct 31 19:12:13.979758 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:13.979729    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"]
Oct 31 19:12:14.020994 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.020957    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ht4p\" (UniqueName: \"kubernetes.io/projected/8788603e-7b27-4f88-aac2-e7ca790925d0-kube-api-access-2ht4p\") pod \"payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2\" (UID: \"8788603e-7b27-4f88-aac2-e7ca790925d0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"
Oct 31 19:12:14.021218 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.021166    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8788603e-7b27-4f88-aac2-e7ca790925d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2\" (UID: \"8788603e-7b27-4f88-aac2-e7ca790925d0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"
Oct 31 19:12:14.026345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.026314    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-d9922f\""
Oct 31 19:12:14.026709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.026684    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-d9922f-dockercfg-zvlns\""
Oct 31 19:12:14.074343 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.074306    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"]
Oct 31 19:12:14.079290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.079257    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"
Oct 31 19:12:14.122287 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.122252    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8788603e-7b27-4f88-aac2-e7ca790925d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2\" (UID: \"8788603e-7b27-4f88-aac2-e7ca790925d0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"
Oct 31 19:12:14.122511 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.122347    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ht4p\" (UniqueName: \"kubernetes.io/projected/8788603e-7b27-4f88-aac2-e7ca790925d0-kube-api-access-2ht4p\") pod \"payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2\" (UID: \"8788603e-7b27-4f88-aac2-e7ca790925d0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"
Oct 31 19:12:14.123056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.123027    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8788603e-7b27-4f88-aac2-e7ca790925d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2\" (UID: \"8788603e-7b27-4f88-aac2-e7ca790925d0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"
Oct 31 19:12:14.130221 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.130189    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"]
Oct 31 19:12:14.180335 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.180263    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ht4p\" (UniqueName: \"kubernetes.io/projected/8788603e-7b27-4f88-aac2-e7ca790925d0-kube-api-access-2ht4p\") pod \"payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2\" (UID: \"8788603e-7b27-4f88-aac2-e7ca790925d0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"
Oct 31 19:12:14.223094 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.223049    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v\" (UID: \"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4\") " pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"
Oct 31 19:12:14.223290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.223189    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzcs\" (UniqueName: \"kubernetes.io/projected/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-kube-api-access-7xzcs\") pod \"payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v\" (UID: \"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4\") " pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"
Oct 31 19:12:14.239590 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.239553    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"
Oct 31 19:12:14.325224 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.324113    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v\" (UID: \"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4\") " pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"
Oct 31 19:12:14.325224 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.324191    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzcs\" (UniqueName: \"kubernetes.io/projected/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-kube-api-access-7xzcs\") pod \"payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v\" (UID: \"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4\") " pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"
Oct 31 19:12:14.325224 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.325178    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v\" (UID: \"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4\") " pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"
Oct 31 19:12:14.382279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.382241    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzcs\" (UniqueName: \"kubernetes.io/projected/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-kube-api-access-7xzcs\") pod \"payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v\" (UID: \"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4\") " pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"
Oct 31 19:12:14.395075 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.395037    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"
Oct 31 19:12:14.617300 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.617263    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"]
Oct 31 19:12:14.618216 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:12:14.618189    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8788603e_7b27_4f88_aac2_e7ca790925d0.slice/crio-41ba7c49b8c02c15505c11a3af9a7178fb725cac506b98b51d35dd6aa4ac336b WatchSource:0}: Error finding container 41ba7c49b8c02c15505c11a3af9a7178fb725cac506b98b51d35dd6aa4ac336b: Status 404 returned error can't find the container with id 41ba7c49b8c02c15505c11a3af9a7178fb725cac506b98b51d35dd6aa4ac336b
Oct 31 19:12:14.770483 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:14.770438    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"]
Oct 31 19:12:14.771040 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:12:14.771005    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ec962b_4a6f_40f7_b07f_2c2e7543e8a4.slice/crio-33dbd33e465777cdea2b03d79c040e88ff6fcc74b6260a1398452f5ed416b1ef WatchSource:0}: Error finding container 33dbd33e465777cdea2b03d79c040e88ff6fcc74b6260a1398452f5ed416b1ef: Status 404 returned error can't find the container with id 33dbd33e465777cdea2b03d79c040e88ff6fcc74b6260a1398452f5ed416b1ef
Oct 31 19:12:15.084663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:15.084567    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2" event={"ID":"8788603e-7b27-4f88-aac2-e7ca790925d0","Type":"ContainerStarted","Data":"8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7"}
Oct 31 19:12:15.084663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:15.084632    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2" event={"ID":"8788603e-7b27-4f88-aac2-e7ca790925d0","Type":"ContainerStarted","Data":"41ba7c49b8c02c15505c11a3af9a7178fb725cac506b98b51d35dd6aa4ac336b"}
Oct 31 19:12:15.084663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:15.084659    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"
Oct 31 19:12:15.086833 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:15.086804    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v" event={"ID":"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4","Type":"ContainerStarted","Data":"b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886"}
Oct 31 19:12:15.087013 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:15.086854    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v" event={"ID":"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4","Type":"ContainerStarted","Data":"33dbd33e465777cdea2b03d79c040e88ff6fcc74b6260a1398452f5ed416b1ef"}
Oct 31 19:12:15.087013 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:15.086971    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"
Oct 31 19:12:15.199823 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:15.199746    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2" podStartSLOduration=2.199731042 podStartE2EDuration="2.199731042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:12:15.197975562 +0000 UTC m=+3081.344425463" watchObservedRunningTime="2026-10-31 19:12:15.199731042 +0000 UTC m=+3081.346180898"
Oct 31 19:12:15.314117 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:15.313160    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v" podStartSLOduration=1.3131392769999999 podStartE2EDuration="1.313139277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:12:15.311832149 +0000 UTC m=+3081.458282015" watchObservedRunningTime="2026-10-31 19:12:15.313139277 +0000 UTC m=+3081.459589142"
Oct 31 19:12:17.071992 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:17.071957    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:17.343464 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:17.343379    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:21.754862 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:21.754817    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:24.565401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:24.565353    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:24.738398 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:24.738363    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:25.469924 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:12:25.469889    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:12:26.096025 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:26.095992    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"
Oct 31 19:12:26.096432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:26.096073    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"
Oct 31 19:12:26.455999 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:26.455902    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:27.743650 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.743618    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"]
Oct 31 19:12:27.747887 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.747870    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.786628 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.786595    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-dshm\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.786828 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.786648    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htnj2\" (UniqueName: \"kubernetes.io/projected/f01e3592-abbe-44cb-bf71-5d3e039d4526-kube-api-access-htnj2\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.786828 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.786675    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-tmp-dir\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.786828 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.786734    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-model-cache\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.786828 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.786782    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01e3592-abbe-44cb-bf71-5d3e039d4526-tls-certs\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.787053 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.786829    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-home\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.805374 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.805339    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"]
Oct 31 19:12:27.850932 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.850884    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-d1ff32\"/\"openshift-service-ca.crt\""
Oct 31 19:12:27.850932 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.850928    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-d1ff32\"/\"default-dockercfg-jl4rd\""
Oct 31 19:12:27.851149 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.850941    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-d1ff32\"/\"auth-test-model-d1ff32-kserve-self-signed-certs\""
Oct 31 19:12:27.851959 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.851943    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-d1ff32\"/\"kube-root-ca.crt\""
Oct 31 19:12:27.864175 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.864147    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:27.888403 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.888382    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01e3592-abbe-44cb-bf71-5d3e039d4526-tls-certs\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.888553 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.888425    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-home\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.888553 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.888549    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-dshm\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.888661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.888580    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnj2\" (UniqueName: \"kubernetes.io/projected/f01e3592-abbe-44cb-bf71-5d3e039d4526-kube-api-access-htnj2\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.888661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.888602    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-tmp-dir\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.888661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.888647    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-model-cache\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.888950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.888926    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-home\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.889045 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.888987    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-tmp-dir\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.889096 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.889041    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-model-cache\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.890986 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.890966    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-dshm\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.891159 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.891142    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01e3592-abbe-44cb-bf71-5d3e039d4526-tls-certs\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:27.946235 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:27.946197    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnj2\" (UniqueName: \"kubernetes.io/projected/f01e3592-abbe-44cb-bf71-5d3e039d4526-kube-api-access-htnj2\") pod \"auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") " pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:28.063271 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:28.063186    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:28.440073 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:28.440043    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"]
Oct 31 19:12:28.440757 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:12:28.440732    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf01e3592_abbe_44cb_bf71_5d3e039d4526.slice/crio-38719961d5d18410f2a8590668104015c715df61f15febd018677d02b1d0ce70 WatchSource:0}: Error finding container 38719961d5d18410f2a8590668104015c715df61f15febd018677d02b1d0ce70: Status 404 returned error can't find the container with id 38719961d5d18410f2a8590668104015c715df61f15febd018677d02b1d0ce70
Oct 31 19:12:29.177547 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:29.177502    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z" event={"ID":"f01e3592-abbe-44cb-bf71-5d3e039d4526","Type":"ContainerStarted","Data":"4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20"}
Oct 31 19:12:29.177989 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:29.177561    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z" event={"ID":"f01e3592-abbe-44cb-bf71-5d3e039d4526","Type":"ContainerStarted","Data":"38719961d5d18410f2a8590668104015c715df61f15febd018677d02b1d0ce70"}
Oct 31 19:12:29.292466 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:29.292409    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z" podStartSLOduration=2.292396982 podStartE2EDuration="2.292396982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:12:29.291776662 +0000 UTC m=+3095.438226526" watchObservedRunningTime="2026-10-31 19:12:29.292396982 +0000 UTC m=+3095.438846837"
Oct 31 19:12:30.063312 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:30.063274    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:31.154453 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:31.154416    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:31.699322 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:31.699290    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"]
Oct 31 19:12:31.699559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:31.699535    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" podUID="8d4422bb-55c9-4ba7-b963-850377365978" containerName="istio-proxy" containerID="cri-o://40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26" gracePeriod=30
Oct 31 19:12:35.810662 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:35.810567    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:36.152108 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:36.152027    4484 patch_prober.go:29] interesting pod/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:12:36.152108 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:36.152085    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" podUID="8d4422bb-55c9-4ba7-b963-850377365978" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:12:36.971536 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:36.971452    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:12:37.096854 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.096815    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-socket\") pod \"8d4422bb-55c9-4ba7-b963-850377365978\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") "
Oct 31 19:12:37.097039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.096874    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d4422bb-55c9-4ba7-b963-850377365978-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4422bb-55c9-4ba7-b963-850377365978-istio-podinfo\") pod \"8d4422bb-55c9-4ba7-b963-850377365978\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") "
Oct 31 19:12:37.097039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.096906    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-certs\") pod \"8d4422bb-55c9-4ba7-b963-850377365978\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") "
Oct 31 19:12:37.097039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.096936    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-kube-api-access-v2qgg\" (UniqueName: \"kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-kube-api-access-v2qgg\") pod \"8d4422bb-55c9-4ba7-b963-850377365978\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") "
Oct 31 19:12:37.097039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097010    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-data\") pod \"8d4422bb-55c9-4ba7-b963-850377365978\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") "
Oct 31 19:12:37.097233 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097051    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-istio-token\" (UniqueName: \"kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-istio-token\") pod \"8d4422bb-55c9-4ba7-b963-850377365978\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") "
Oct 31 19:12:37.097233 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097088    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4422bb-55c9-4ba7-b963-850377365978-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4422bb-55c9-4ba7-b963-850377365978-istiod-ca-cert\") pod \"8d4422bb-55c9-4ba7-b963-850377365978\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") "
Oct 31 19:12:37.097233 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097142    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-credential-socket\") pod \"8d4422bb-55c9-4ba7-b963-850377365978\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") "
Oct 31 19:12:37.097233 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097155    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-socket" pod "8d4422bb-55c9-4ba7-b963-850377365978" (UID: "8d4422bb-55c9-4ba7-b963-850377365978"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:37.097501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097288    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-envoy\") pod \"8d4422bb-55c9-4ba7-b963-850377365978\" (UID: \"8d4422bb-55c9-4ba7-b963-850377365978\") "
Oct 31 19:12:37.097501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097299    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-certs" pod "8d4422bb-55c9-4ba7-b963-850377365978" (UID: "8d4422bb-55c9-4ba7-b963-850377365978"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:37.097501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097415    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-data" pod "8d4422bb-55c9-4ba7-b963-850377365978" (UID: "8d4422bb-55c9-4ba7-b963-850377365978"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:37.097685 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097647    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-credential-socket" pod "8d4422bb-55c9-4ba7-b963-850377365978" (UID: "8d4422bb-55c9-4ba7-b963-850377365978"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:37.097685 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097554    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4422bb-55c9-4ba7-b963-850377365978-istiod-ca-cert" pod "8d4422bb-55c9-4ba7-b963-850377365978" (UID: "8d4422bb-55c9-4ba7-b963-850377365978"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:12:37.097980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097909    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4422bb-55c9-4ba7-b963-850377365978-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:37.097980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097936    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:37.097980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097953    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:37.097980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097966    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:37.097980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.097980    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:37.099489 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.099450    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d4422bb-55c9-4ba7-b963-850377365978-istio-podinfo" pod "8d4422bb-55c9-4ba7-b963-850377365978" (UID: "8d4422bb-55c9-4ba7-b963-850377365978"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:12:37.099607 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.099568    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-kube-api-access-v2qgg" pod "8d4422bb-55c9-4ba7-b963-850377365978" (UID: "8d4422bb-55c9-4ba7-b963-850377365978"). InnerVolumeSpecName "kube-api-access-v2qgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:12:37.099607 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.099571    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-envoy" pod "8d4422bb-55c9-4ba7-b963-850377365978" (UID: "8d4422bb-55c9-4ba7-b963-850377365978"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:12:37.099768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.099754    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-istio-token" pod "8d4422bb-55c9-4ba7-b963-850377365978" (UID: "8d4422bb-55c9-4ba7-b963-850377365978"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:12:37.199060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.199027    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:37.199060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.199055    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4422bb-55c9-4ba7-b963-850377365978-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:37.199060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.199064    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4422bb-55c9-4ba7-b963-850377365978-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:37.199303 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.199073    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2qgg\" (UniqueName: \"kubernetes.io/projected/8d4422bb-55c9-4ba7-b963-850377365978-kube-api-access-v2qgg\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:12:37.236194 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.236170    4484 generic.go:356] "Generic (PLEG): container finished" podID="8d4422bb-55c9-4ba7-b963-850377365978" containerID="40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26" exitCode=0
Oct 31 19:12:37.236420 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.236220    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"
Oct 31 19:12:37.236420 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.236264    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" event={"ID":"8d4422bb-55c9-4ba7-b963-850377365978","Type":"ContainerDied","Data":"40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26"}
Oct 31 19:12:37.236420 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.236319    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb" event={"ID":"8d4422bb-55c9-4ba7-b963-850377365978","Type":"ContainerDied","Data":"37756a208de787537b17435f7026924a9a7e39cc1f481162f517d31666fa3f25"}
Oct 31 19:12:37.236420 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.236346    4484 scope.go:122] "RemoveContainer" containerID="40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26"
Oct 31 19:12:37.248172 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.248147    4484 scope.go:122] "RemoveContainer" containerID="40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26"
Oct 31 19:12:37.248711 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:12:37.248674    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26\": container with ID starting with 40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26 not found: ID does not exist" containerID="40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26"
Oct 31 19:12:37.248795 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.248719    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26"} err="failed to get container status \"40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26\": rpc error: code = NotFound desc = could not find container \"40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26\": container with ID starting with 40a730e8c7846abe4764920a9e0a44d585d4b695a5dc4aa482bde18544790f26 not found: ID does not exist"
Oct 31 19:12:37.406292 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.406255    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"]
Oct 31 19:12:37.410077 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.410053    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-q7fcb"]
Oct 31 19:12:37.994632 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:37.994601    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4422bb-55c9-4ba7-b963-850377365978" path="/var/lib/kubelet/pods/8d4422bb-55c9-4ba7-b963-850377365978/volumes"
Oct 31 19:12:38.063294 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:38.063259    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:38.076392 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:38.076362    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:38.255958 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:38.255866    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:12:40.437122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:40.437086    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:41.463044 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:41.463010    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:12:45.059613 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:45.059564    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:49.752970 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:49.752930    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:54.365842 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:54.365802    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:55.469928 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:12:55.469891    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:12:57.812889 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:57.812840    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:12:58.967952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:12:58.967908    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:02.207990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:02.207949    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:03.590899 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:03.590846    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:08.251628 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:08.251567    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:12.855862 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:12.855821    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:17.214267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.214230    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"]
Oct 31 19:13:17.215230 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.215210    4484 cpu_manager.go:414] "containerMap: removing container" podUID="8d4422bb-55c9-4ba7-b963-850377365978" containerName="istio-proxy"
Oct 31 19:13:17.215286 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.215235    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4422bb-55c9-4ba7-b963-850377365978" containerName="istio-proxy"
Oct 31 19:13:17.215439 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.215428    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4422bb-55c9-4ba7-b963-850377365978" containerName="istio-proxy"
Oct 31 19:13:17.220849 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.220803    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.273960 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.273932    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"]
Oct 31 19:13:17.294651 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.294610    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-metrics-certs\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.294839 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.294671    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-maas-api-tls\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.294839 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.294784    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05de422c-b712-4e93-af92-d42732e69015-openshift-service-ca\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.294839 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.294832    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k76t4\" (UniqueName: \"kubernetes.io/projected/05de422c-b712-4e93-af92-d42732e69015-kube-api-access-k76t4\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.321112 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.320996    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-d9922f-serving-cert\""
Oct 31 19:13:17.395853 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.395822    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-metrics-certs\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.396059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.395864    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-maas-api-tls\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.396059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.395930    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05de422c-b712-4e93-af92-d42732e69015-openshift-service-ca\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.396059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.395966    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k76t4\" (UniqueName: \"kubernetes.io/projected/05de422c-b712-4e93-af92-d42732e69015-kube-api-access-k76t4\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.396255 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:13:17.396055    4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-serving-cert: secret "maas-api-e2e-explicit-d9922f-serving-cert" not found
Oct 31 19:13:17.396255 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:13:17.396135    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-maas-api-tls podName:05de422c-b712-4e93-af92-d42732e69015 nodeName:}" failed. No retries permitted until 2026-10-31 19:13:17.896114767 +0000 UTC m=+3144.042564638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-maas-api-tls") pod "maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg" (UID: "05de422c-b712-4e93-af92-d42732e69015") : secret "maas-api-e2e-explicit-d9922f-serving-cert" not found
Oct 31 19:13:17.396764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.396740    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05de422c-b712-4e93-af92-d42732e69015-openshift-service-ca\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.398689 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.398668    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-metrics-certs\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.418834 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.418801    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:17.455919 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.455886    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k76t4\" (UniqueName: \"kubernetes.io/projected/05de422c-b712-4e93-af92-d42732e69015-kube-api-access-k76t4\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.585599 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.585565    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:17.768933 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.768892    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:17.903018 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.902922    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-maas-api-tls\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:17.905568 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:17.905541    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-maas-api-tls\") pod \"maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:18.135201 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:18.135166    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:18.514248 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:18.514219    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"]
Oct 31 19:13:18.514737 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:13:18.514714    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05de422c_b712_4e93_af92_d42732e69015.slice/crio-58e6b55495c95638aff9563225ebfd4e6ac0f8d6383e5b83ea606fcf0ce0346d WatchSource:0}: Error finding container 58e6b55495c95638aff9563225ebfd4e6ac0f8d6383e5b83ea606fcf0ce0346d: Status 404 returned error can't find the container with id 58e6b55495c95638aff9563225ebfd4e6ac0f8d6383e5b83ea606fcf0ce0346d
Oct 31 19:13:18.515958 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:18.515940    4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:13:19.515258 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:19.515221    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg" event={"ID":"05de422c-b712-4e93-af92-d42732e69015","Type":"ContainerStarted","Data":"28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd"}
Oct 31 19:13:19.515667 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:19.515284    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg" event={"ID":"05de422c-b712-4e93-af92-d42732e69015","Type":"ContainerStarted","Data":"58e6b55495c95638aff9563225ebfd4e6ac0f8d6383e5b83ea606fcf0ce0346d"}
Oct 31 19:13:19.515667 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:19.515324    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:19.634204 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:19.634155    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg" podStartSLOduration=1.750592497 podStartE2EDuration="2.634140362s" totalImagesPullingTime="883.547865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:13:19.631520065 +0000 UTC m=+3145.777969932" watchObservedRunningTime="2026-10-31 19:13:19.634140362 +0000 UTC m=+3145.780590206"
Oct 31 19:13:22.289771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:22.289740    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:25.469729 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:13:25.469696    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:13:25.527435 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:25.527410    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:13:26.903602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:26.903568    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:31.538721 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:31.538677    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:36.150117 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:36.150081    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:39.723660 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:39.723624    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:13:40.472935 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:40.472894    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:40.901814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:40.901757    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:45.548651 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:45.548605    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:50.192119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:50.192087    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:54.843206 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:54.843169    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:55.469309 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:13:55.469267    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:13:58.145094 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:58.145059    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:13:59.467952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:13:59.467915    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:02.541566 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:02.541533    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:04.121342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:04.121301    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:08.787787 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:08.787701    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:13.432030 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:13.431986    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:17.014317 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:17.014277    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:14:18.165420 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:18.165382    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:22.855202 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:22.855157    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:25.468837 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:14:25.468799    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:14:27.557404 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:27.557366    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:32.250085 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:32.250046    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:33.315174 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:33.315144    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:35.652097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:35.652066    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:37.263534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:37.263468    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:41.955100 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:41.955067    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:46.590329 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:46.590282    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:51.260935 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:51.260901    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:55.468379 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:14:55.468343    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:14:55.898917 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:55.898877    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:57.211268 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:57.211228    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:14:58.484698 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:14:58.484666    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:00.166563 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.166529    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"]
Oct 31 19:15:00.170947 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.170928    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"
Oct 31 19:15:00.192286 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.192260    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"]
Oct 31 19:15:00.196309 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.196293    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"
Oct 31 19:15:00.208371 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.208347    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"]
Oct 31 19:15:00.212513 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.212488    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"
Oct 31 19:15:00.217043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.217019    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"]
Oct 31 19:15:00.221235 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.221216    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"
Oct 31 19:15:00.239783 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.239761    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:15:00.261175 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.261152    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"]
Oct 31 19:15:00.315035 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.314992    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"]
Oct 31 19:15:00.320244 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.320207    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83c0e54-ebb9-4621-ab5a-f95010eb9158-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6\" (UID: \"e83c0e54-ebb9-4621-ab5a-f95010eb9158\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"
Oct 31 19:15:00.320428 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.320363    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/440ed2aa-0722-4e4a-9434-24a93962abd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv\" (UID: \"440ed2aa-0722-4e4a-9434-24a93962abd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"
Oct 31 19:15:00.320428 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.320410    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf8g4\" (UniqueName: \"kubernetes.io/projected/e83c0e54-ebb9-4621-ab5a-f95010eb9158-kube-api-access-qf8g4\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6\" (UID: \"e83c0e54-ebb9-4621-ab5a-f95010eb9158\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"
Oct 31 19:15:00.320599 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.320440    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc2126d-aec4-4cff-983b-2c760030227c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8\" (UID: \"3bc2126d-aec4-4cff-983b-2c760030227c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"
Oct 31 19:15:00.320599 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.320520    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztbz\" (UniqueName: \"kubernetes.io/projected/3bc2126d-aec4-4cff-983b-2c760030227c-kube-api-access-6ztbz\") pod \"maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8\" (UID: \"3bc2126d-aec4-4cff-983b-2c760030227c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"
Oct 31 19:15:00.320599 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.320559    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aefadc-e01e-4c43-92ee-94f11332c501-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq\" (UID: \"a9aefadc-e01e-4c43-92ee-94f11332c501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"
Oct 31 19:15:00.320599 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.320577    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kgwl\" (UniqueName: \"kubernetes.io/projected/a9aefadc-e01e-4c43-92ee-94f11332c501-kube-api-access-2kgwl\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq\" (UID: \"a9aefadc-e01e-4c43-92ee-94f11332c501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"
Oct 31 19:15:00.320599 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.320596    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsvx7\" (UniqueName: \"kubernetes.io/projected/440ed2aa-0722-4e4a-9434-24a93962abd1-kube-api-access-qsvx7\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv\" (UID: \"440ed2aa-0722-4e4a-9434-24a93962abd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"
Oct 31 19:15:00.352924 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.352895    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"]
Oct 31 19:15:00.363853 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.363820    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"]
Oct 31 19:15:00.422141 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.422051    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aefadc-e01e-4c43-92ee-94f11332c501-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq\" (UID: \"a9aefadc-e01e-4c43-92ee-94f11332c501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"
Oct 31 19:15:00.422141 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.422089    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kgwl\" (UniqueName: \"kubernetes.io/projected/a9aefadc-e01e-4c43-92ee-94f11332c501-kube-api-access-2kgwl\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq\" (UID: \"a9aefadc-e01e-4c43-92ee-94f11332c501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"
Oct 31 19:15:00.422141 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.422110    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsvx7\" (UniqueName: \"kubernetes.io/projected/440ed2aa-0722-4e4a-9434-24a93962abd1-kube-api-access-qsvx7\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv\" (UID: \"440ed2aa-0722-4e4a-9434-24a93962abd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"
Oct 31 19:15:00.422961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.422181    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83c0e54-ebb9-4621-ab5a-f95010eb9158-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6\" (UID: \"e83c0e54-ebb9-4621-ab5a-f95010eb9158\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"
Oct 31 19:15:00.422961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.422231    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/440ed2aa-0722-4e4a-9434-24a93962abd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv\" (UID: \"440ed2aa-0722-4e4a-9434-24a93962abd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"
Oct 31 19:15:00.422961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.422257    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf8g4\" (UniqueName: \"kubernetes.io/projected/e83c0e54-ebb9-4621-ab5a-f95010eb9158-kube-api-access-qf8g4\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6\" (UID: \"e83c0e54-ebb9-4621-ab5a-f95010eb9158\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"
Oct 31 19:15:00.422961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.422278    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc2126d-aec4-4cff-983b-2c760030227c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8\" (UID: \"3bc2126d-aec4-4cff-983b-2c760030227c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"
Oct 31 19:15:00.422961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.422348    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztbz\" (UniqueName: \"kubernetes.io/projected/3bc2126d-aec4-4cff-983b-2c760030227c-kube-api-access-6ztbz\") pod \"maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8\" (UID: \"3bc2126d-aec4-4cff-983b-2c760030227c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"
Oct 31 19:15:00.422961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.422914    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aefadc-e01e-4c43-92ee-94f11332c501-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq\" (UID: \"a9aefadc-e01e-4c43-92ee-94f11332c501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"
Oct 31 19:15:00.423206 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.422995    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/440ed2aa-0722-4e4a-9434-24a93962abd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv\" (UID: \"440ed2aa-0722-4e4a-9434-24a93962abd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"
Oct 31 19:15:00.423206 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.423065    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc2126d-aec4-4cff-983b-2c760030227c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8\" (UID: \"3bc2126d-aec4-4cff-983b-2c760030227c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"
Oct 31 19:15:00.423206 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.423071    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83c0e54-ebb9-4621-ab5a-f95010eb9158-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6\" (UID: \"e83c0e54-ebb9-4621-ab5a-f95010eb9158\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"
Oct 31 19:15:00.479135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.479101    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kgwl\" (UniqueName: \"kubernetes.io/projected/a9aefadc-e01e-4c43-92ee-94f11332c501-kube-api-access-2kgwl\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq\" (UID: \"a9aefadc-e01e-4c43-92ee-94f11332c501\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"
Oct 31 19:15:00.479413 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.479392    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsvx7\" (UniqueName: \"kubernetes.io/projected/440ed2aa-0722-4e4a-9434-24a93962abd1-kube-api-access-qsvx7\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv\" (UID: \"440ed2aa-0722-4e4a-9434-24a93962abd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"
Oct 31 19:15:00.479528 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.479457    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf8g4\" (UniqueName: \"kubernetes.io/projected/e83c0e54-ebb9-4621-ab5a-f95010eb9158-kube-api-access-qf8g4\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6\" (UID: \"e83c0e54-ebb9-4621-ab5a-f95010eb9158\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"
Oct 31 19:15:00.479528 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.479512    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztbz\" (UniqueName: \"kubernetes.io/projected/3bc2126d-aec4-4cff-983b-2c760030227c-kube-api-access-6ztbz\") pod \"maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8\" (UID: \"3bc2126d-aec4-4cff-983b-2c760030227c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"
Oct 31 19:15:00.487336 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.487321    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"
Oct 31 19:15:00.512579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.512555    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"
Oct 31 19:15:00.526374 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.526157    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"
Oct 31 19:15:00.535626 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.535597    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"
Oct 31 19:15:00.651979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.651949    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:00.874331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.874298    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"]
Oct 31 19:15:00.874674 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:15:00.874647    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83c0e54_ebb9_4621_ab5a_f95010eb9158.slice/crio-1a0f0773f21425962f34009e0ce71a12e2b1b79bd68965d3bf4a270caa1deefb WatchSource:0}: Error finding container 1a0f0773f21425962f34009e0ce71a12e2b1b79bd68965d3bf4a270caa1deefb: Status 404 returned error can't find the container with id 1a0f0773f21425962f34009e0ce71a12e2b1b79bd68965d3bf4a270caa1deefb
Oct 31 19:15:00.921555 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:15:00.921528    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440ed2aa_0722_4e4a_9434_24a93962abd1.slice/crio-40bf25b4b1f4a5e5cce6e408d1be7d133f877f423ab570a26b8a016b7cc4ca24 WatchSource:0}: Error finding container 40bf25b4b1f4a5e5cce6e408d1be7d133f877f423ab570a26b8a016b7cc4ca24: Status 404 returned error can't find the container with id 40bf25b4b1f4a5e5cce6e408d1be7d133f877f423ab570a26b8a016b7cc4ca24
Oct 31 19:15:00.921555 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.921540    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"]
Oct 31 19:15:00.969288 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.969268    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"]
Oct 31 19:15:00.972261 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:00.972239    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"]
Oct 31 19:15:01.064043 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:15:01.064006    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9aefadc_e01e_4c43_92ee_94f11332c501.slice/crio-2acbc0a19dcf68effe8d9b9c5bba77275e45ab1c0da36e43fe56e83ad5e90c4d WatchSource:0}: Error finding container 2acbc0a19dcf68effe8d9b9c5bba77275e45ab1c0da36e43fe56e83ad5e90c4d: Status 404 returned error can't find the container with id 2acbc0a19dcf68effe8d9b9c5bba77275e45ab1c0da36e43fe56e83ad5e90c4d
Oct 31 19:15:01.065382 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:15:01.065098    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc2126d_aec4_4cff_983b_2c760030227c.slice/crio-68e8be6edd63bc725de1b777e5989f094f3e9e284b38a055444685114cf513c2 WatchSource:0}: Error finding container 68e8be6edd63bc725de1b777e5989f094f3e9e284b38a055444685114cf513c2: Status 404 returned error can't find the container with id 68e8be6edd63bc725de1b777e5989f094f3e9e284b38a055444685114cf513c2
Oct 31 19:15:01.194381 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:01.194346    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq" event={"ID":"a9aefadc-e01e-4c43-92ee-94f11332c501","Type":"ContainerStarted","Data":"2acbc0a19dcf68effe8d9b9c5bba77275e45ab1c0da36e43fe56e83ad5e90c4d"}
Oct 31 19:15:01.195958 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:01.195929    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6" event={"ID":"e83c0e54-ebb9-4621-ab5a-f95010eb9158","Type":"ContainerStarted","Data":"bcf96cff2ad335c5f0df22f445737afe2128a3e65ab1409ee777375d06644c05"}
Oct 31 19:15:01.196091 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:01.195981    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6" event={"ID":"e83c0e54-ebb9-4621-ab5a-f95010eb9158","Type":"ContainerStarted","Data":"1a0f0773f21425962f34009e0ce71a12e2b1b79bd68965d3bf4a270caa1deefb"}
Oct 31 19:15:01.197086 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:01.197064    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv" event={"ID":"440ed2aa-0722-4e4a-9434-24a93962abd1","Type":"ContainerStarted","Data":"40bf25b4b1f4a5e5cce6e408d1be7d133f877f423ab570a26b8a016b7cc4ca24"}
Oct 31 19:15:01.198289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:01.198269    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8" event={"ID":"3bc2126d-aec4-4cff-983b-2c760030227c","Type":"ContainerStarted","Data":"68e8be6edd63bc725de1b777e5989f094f3e9e284b38a055444685114cf513c2"}
Oct 31 19:15:01.312488 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:01.312432    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6" podStartSLOduration=1.312415233 podStartE2EDuration="1.312415233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:01.309671304 +0000 UTC m=+3247.456121181" watchObservedRunningTime="2026-10-31 19:15:01.312415233 +0000 UTC m=+3247.458865139"
Oct 31 19:15:02.207052 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.207024    4484 generic.go:356] "Generic (PLEG): container finished" podID="440ed2aa-0722-4e4a-9434-24a93962abd1" containerID="0546e4a942e6bd260f071acc82e910e3253da9ed2a9ed2728e9a624f3a4b6e84" exitCode=60
Oct 31 19:15:02.207514 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.207103    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv" event={"ID":"440ed2aa-0722-4e4a-9434-24a93962abd1","Type":"ContainerDied","Data":"0546e4a942e6bd260f071acc82e910e3253da9ed2a9ed2728e9a624f3a4b6e84"}
Oct 31 19:15:02.207514 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.207398    4484 scope.go:122] "RemoveContainer" containerID="0546e4a942e6bd260f071acc82e910e3253da9ed2a9ed2728e9a624f3a4b6e84"
Oct 31 19:15:02.209097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.208979    4484 generic.go:356] "Generic (PLEG): container finished" podID="3bc2126d-aec4-4cff-983b-2c760030227c" containerID="cfcb0d643557bb3932e54d4cecb80f2f67c43c8bb92d8e23f638ac947e160e8e" exitCode=60
Oct 31 19:15:02.209097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.209016    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8" event={"ID":"3bc2126d-aec4-4cff-983b-2c760030227c","Type":"ContainerDied","Data":"cfcb0d643557bb3932e54d4cecb80f2f67c43c8bb92d8e23f638ac947e160e8e"}
Oct 31 19:15:02.209334 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.209315    4484 scope.go:122] "RemoveContainer" containerID="cfcb0d643557bb3932e54d4cecb80f2f67c43c8bb92d8e23f638ac947e160e8e"
Oct 31 19:15:02.211012 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.210829    4484 generic.go:356] "Generic (PLEG): container finished" podID="a9aefadc-e01e-4c43-92ee-94f11332c501" containerID="c202e26f2eccc99b2005b4325ec0408dfa6370d3e4d4ee8fda1129bb6b8896aa" exitCode=60
Oct 31 19:15:02.211114 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.211084    4484 scope.go:122] "RemoveContainer" containerID="c202e26f2eccc99b2005b4325ec0408dfa6370d3e4d4ee8fda1129bb6b8896aa"
Oct 31 19:15:02.211212 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.211137    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq" event={"ID":"a9aefadc-e01e-4c43-92ee-94f11332c501","Type":"ContainerDied","Data":"c202e26f2eccc99b2005b4325ec0408dfa6370d3e4d4ee8fda1129bb6b8896aa"}
Oct 31 19:15:02.212908 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.212890    4484 generic.go:356] "Generic (PLEG): container finished" podID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" containerID="bcf96cff2ad335c5f0df22f445737afe2128a3e65ab1409ee777375d06644c05" exitCode=60
Oct 31 19:15:02.212987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.212959    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6" event={"ID":"e83c0e54-ebb9-4621-ab5a-f95010eb9158","Type":"ContainerDied","Data":"bcf96cff2ad335c5f0df22f445737afe2128a3e65ab1409ee777375d06644c05"}
Oct 31 19:15:02.213112 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.213099    4484 scope.go:122] "RemoveContainer" containerID="bcf96cff2ad335c5f0df22f445737afe2128a3e65ab1409ee777375d06644c05"
Oct 31 19:15:02.943210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:02.943116    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:03.221804 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:03.221739    4484 generic.go:356] "Generic (PLEG): container finished" podID="440ed2aa-0722-4e4a-9434-24a93962abd1" containerID="4d42ff8c98f6f79ae796e89ae55797902d6ce5afefdc9232b17a42126afdbc16" exitCode=60
Oct 31 19:15:03.221804 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:03.221787    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv" event={"ID":"440ed2aa-0722-4e4a-9434-24a93962abd1","Type":"ContainerDied","Data":"4d42ff8c98f6f79ae796e89ae55797902d6ce5afefdc9232b17a42126afdbc16"}
Oct 31 19:15:03.222278 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:03.221853    4484 scope.go:122] "RemoveContainer" containerID="0546e4a942e6bd260f071acc82e910e3253da9ed2a9ed2728e9a624f3a4b6e84"
Oct 31 19:15:03.222278 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:03.222106    4484 scope.go:122] "RemoveContainer" containerID="4d42ff8c98f6f79ae796e89ae55797902d6ce5afefdc9232b17a42126afdbc16"
Oct 31 19:15:03.222439 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:03.222420    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv_odh-ai-gateway-infra(440ed2aa-0722-4e4a-9434-24a93962abd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1"
Oct 31 19:15:03.224036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:03.224008    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8" event={"ID":"3bc2126d-aec4-4cff-983b-2c760030227c","Type":"ContainerStarted","Data":"74879f929ef38681cb57abb63a46d908f6d6c236d35a283f61daee84b11f6c19"}
Oct 31 19:15:03.225922 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:03.225891    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq" event={"ID":"a9aefadc-e01e-4c43-92ee-94f11332c501","Type":"ContainerStarted","Data":"dbaa96dbfdb2fd50d48c85e025951fb967f50e1b1eaec5aa21097b0dda7100fe"}
Oct 31 19:15:03.227990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:03.227970    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6" event={"ID":"e83c0e54-ebb9-4621-ab5a-f95010eb9158","Type":"ContainerStarted","Data":"b8a81d967eb1564c2fa47eb6096b2192fccbae8ac62c31b10d848b1ff16ad5ae"}
Oct 31 19:15:03.423528 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:03.423493    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:15:03.450346 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:03.450299    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq" podStartSLOduration=3.450286057 podStartE2EDuration="3.450286057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:03.449709966 +0000 UTC m=+3249.596159833" watchObservedRunningTime="2026-10-31 19:15:03.450286057 +0000 UTC m=+3249.596735919"
Oct 31 19:15:03.566443 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:03.566380    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8" podStartSLOduration=3.566361422 podStartE2EDuration="3.566361422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:03.56340313 +0000 UTC m=+3249.709853009" watchObservedRunningTime="2026-10-31 19:15:03.566361422 +0000 UTC m=+3249.712811288"
Oct 31 19:15:04.237438 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.237412    4484 scope.go:122] "RemoveContainer" containerID="4d42ff8c98f6f79ae796e89ae55797902d6ce5afefdc9232b17a42126afdbc16"
Oct 31 19:15:04.237981 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:04.237767    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv_odh-ai-gateway-infra(440ed2aa-0722-4e4a-9434-24a93962abd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1"
Oct 31 19:15:04.239330 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.239305    4484 generic.go:356] "Generic (PLEG): container finished" podID="3bc2126d-aec4-4cff-983b-2c760030227c" containerID="74879f929ef38681cb57abb63a46d908f6d6c236d35a283f61daee84b11f6c19" exitCode=60
Oct 31 19:15:04.239461 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.239382    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8" event={"ID":"3bc2126d-aec4-4cff-983b-2c760030227c","Type":"ContainerDied","Data":"74879f929ef38681cb57abb63a46d908f6d6c236d35a283f61daee84b11f6c19"}
Oct 31 19:15:04.239461 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.239448    4484 scope.go:122] "RemoveContainer" containerID="cfcb0d643557bb3932e54d4cecb80f2f67c43c8bb92d8e23f638ac947e160e8e"
Oct 31 19:15:04.239652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.239634    4484 scope.go:122] "RemoveContainer" containerID="74879f929ef38681cb57abb63a46d908f6d6c236d35a283f61daee84b11f6c19"
Oct 31 19:15:04.239944 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:04.239926    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8_odh-ai-gateway-infra(3bc2126d-aec4-4cff-983b-2c760030227c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8" podUID="3bc2126d-aec4-4cff-983b-2c760030227c"
Oct 31 19:15:04.241617 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.241596    4484 generic.go:356] "Generic (PLEG): container finished" podID="a9aefadc-e01e-4c43-92ee-94f11332c501" containerID="dbaa96dbfdb2fd50d48c85e025951fb967f50e1b1eaec5aa21097b0dda7100fe" exitCode=60
Oct 31 19:15:04.241709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.241670    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq" event={"ID":"a9aefadc-e01e-4c43-92ee-94f11332c501","Type":"ContainerDied","Data":"dbaa96dbfdb2fd50d48c85e025951fb967f50e1b1eaec5aa21097b0dda7100fe"}
Oct 31 19:15:04.241884 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.241870    4484 scope.go:122] "RemoveContainer" containerID="dbaa96dbfdb2fd50d48c85e025951fb967f50e1b1eaec5aa21097b0dda7100fe"
Oct 31 19:15:04.242130 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:04.242109    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq_odh-ai-gateway-infra(a9aefadc-e01e-4c43-92ee-94f11332c501)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501"
Oct 31 19:15:04.243714 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.243694    4484 generic.go:356] "Generic (PLEG): container finished" podID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" containerID="b8a81d967eb1564c2fa47eb6096b2192fccbae8ac62c31b10d848b1ff16ad5ae" exitCode=60
Oct 31 19:15:04.243818 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.243740    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6" event={"ID":"e83c0e54-ebb9-4621-ab5a-f95010eb9158","Type":"ContainerDied","Data":"b8a81d967eb1564c2fa47eb6096b2192fccbae8ac62c31b10d848b1ff16ad5ae"}
Oct 31 19:15:04.243946 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.243934    4484 scope.go:122] "RemoveContainer" containerID="b8a81d967eb1564c2fa47eb6096b2192fccbae8ac62c31b10d848b1ff16ad5ae"
Oct 31 19:15:04.244219 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:04.244131    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6_odh-ai-gateway-infra(e83c0e54-ebb9-4621-ab5a-f95010eb9158)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158"
Oct 31 19:15:04.252844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.252818    4484 scope.go:122] "RemoveContainer" containerID="c202e26f2eccc99b2005b4325ec0408dfa6370d3e4d4ee8fda1129bb6b8896aa"
Oct 31 19:15:04.265867 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:04.265843    4484 scope.go:122] "RemoveContainer" containerID="bcf96cff2ad335c5f0df22f445737afe2128a3e65ab1409ee777375d06644c05"
Oct 31 19:15:05.252083 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:05.252055    4484 scope.go:122] "RemoveContainer" containerID="74879f929ef38681cb57abb63a46d908f6d6c236d35a283f61daee84b11f6c19"
Oct 31 19:15:05.252576 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:05.252339    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8_odh-ai-gateway-infra(3bc2126d-aec4-4cff-983b-2c760030227c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8" podUID="3bc2126d-aec4-4cff-983b-2c760030227c"
Oct 31 19:15:05.253794 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:05.253771    4484 scope.go:122] "RemoveContainer" containerID="dbaa96dbfdb2fd50d48c85e025951fb967f50e1b1eaec5aa21097b0dda7100fe"
Oct 31 19:15:05.254058 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:05.254039    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq_odh-ai-gateway-infra(a9aefadc-e01e-4c43-92ee-94f11332c501)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501"
Oct 31 19:15:05.255407 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:05.255387    4484 scope.go:122] "RemoveContainer" containerID="b8a81d967eb1564c2fa47eb6096b2192fccbae8ac62c31b10d848b1ff16ad5ae"
Oct 31 19:15:05.255649 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:05.255632    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6_odh-ai-gateway-infra(e83c0e54-ebb9-4621-ab5a-f95010eb9158)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158"
Oct 31 19:15:05.365036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:05.364997    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:10.053785 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:10.053743    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:14.758850 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:14.758808    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:14.986520 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:14.986492    4484 scope.go:122] "RemoveContainer" containerID="dbaa96dbfdb2fd50d48c85e025951fb967f50e1b1eaec5aa21097b0dda7100fe"
Oct 31 19:15:14.986721 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:14.986707    4484 scope.go:122] "RemoveContainer" containerID="b8a81d967eb1564c2fa47eb6096b2192fccbae8ac62c31b10d848b1ff16ad5ae"
Oct 31 19:15:15.986633 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:15.986611    4484 scope.go:122] "RemoveContainer" containerID="74879f929ef38681cb57abb63a46d908f6d6c236d35a283f61daee84b11f6c19"
Oct 31 19:15:16.333666 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:16.333582    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8" event={"ID":"3bc2126d-aec4-4cff-983b-2c760030227c","Type":"ContainerStarted","Data":"9a02a2ce30431b751480e2be74d988f8af3a2d7f92b97ed7577af5988a0bcf31"}
Oct 31 19:15:16.335645 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:16.335623    4484 generic.go:356] "Generic (PLEG): container finished" podID="a9aefadc-e01e-4c43-92ee-94f11332c501" containerID="622fff241a24e9214385a5321f716c5c3eb555e06c510b38a649ce4d925868d9" exitCode=60
Oct 31 19:15:16.335776 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:16.335694    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq" event={"ID":"a9aefadc-e01e-4c43-92ee-94f11332c501","Type":"ContainerDied","Data":"622fff241a24e9214385a5321f716c5c3eb555e06c510b38a649ce4d925868d9"}
Oct 31 19:15:16.335776 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:16.335742    4484 scope.go:122] "RemoveContainer" containerID="dbaa96dbfdb2fd50d48c85e025951fb967f50e1b1eaec5aa21097b0dda7100fe"
Oct 31 19:15:16.336130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:16.336101    4484 scope.go:122] "RemoveContainer" containerID="622fff241a24e9214385a5321f716c5c3eb555e06c510b38a649ce4d925868d9"
Oct 31 19:15:16.336424 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:16.336387    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq_odh-ai-gateway-infra(a9aefadc-e01e-4c43-92ee-94f11332c501)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501"
Oct 31 19:15:16.342069 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:16.342040    4484 generic.go:356] "Generic (PLEG): container finished" podID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" containerID="7706c623e607070b8faf1571be46e5dee6b347e8b9825ee3404fe9c5af1e058d" exitCode=60
Oct 31 19:15:16.342223 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:16.342156    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6" event={"ID":"e83c0e54-ebb9-4621-ab5a-f95010eb9158","Type":"ContainerDied","Data":"7706c623e607070b8faf1571be46e5dee6b347e8b9825ee3404fe9c5af1e058d"}
Oct 31 19:15:16.342538 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:16.342519    4484 scope.go:122] "RemoveContainer" containerID="7706c623e607070b8faf1571be46e5dee6b347e8b9825ee3404fe9c5af1e058d"
Oct 31 19:15:16.342800 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:16.342782    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6_odh-ai-gateway-infra(e83c0e54-ebb9-4621-ab5a-f95010eb9158)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158"
Oct 31 19:15:16.349693 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:16.349668    4484 scope.go:122] "RemoveContainer" containerID="b8a81d967eb1564c2fa47eb6096b2192fccbae8ac62c31b10d848b1ff16ad5ae"
Oct 31 19:15:17.350234 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.350199    4484 generic.go:356] "Generic (PLEG): container finished" podID="3bc2126d-aec4-4cff-983b-2c760030227c" containerID="9a02a2ce30431b751480e2be74d988f8af3a2d7f92b97ed7577af5988a0bcf31" exitCode=60
Oct 31 19:15:17.350718 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.350277    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8" event={"ID":"3bc2126d-aec4-4cff-983b-2c760030227c","Type":"ContainerDied","Data":"9a02a2ce30431b751480e2be74d988f8af3a2d7f92b97ed7577af5988a0bcf31"}
Oct 31 19:15:17.350718 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.350355    4484 scope.go:122] "RemoveContainer" containerID="74879f929ef38681cb57abb63a46d908f6d6c236d35a283f61daee84b11f6c19"
Oct 31 19:15:17.350854 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.350777    4484 scope.go:122] "RemoveContainer" containerID="9a02a2ce30431b751480e2be74d988f8af3a2d7f92b97ed7577af5988a0bcf31"
Oct 31 19:15:17.351084 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:17.351061    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8_odh-ai-gateway-infra(3bc2126d-aec4-4cff-983b-2c760030227c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8" podUID="3bc2126d-aec4-4cff-983b-2c760030227c"
Oct 31 19:15:17.459514 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.459461    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"]
Oct 31 19:15:17.573696 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.573659    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"]
Oct 31 19:15:17.686306 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.686274    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"]
Oct 31 19:15:17.743314 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.743290    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"
Oct 31 19:15:17.822755 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.822723    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e83c0e54-ebb9-4621-ab5a-f95010eb9158-kube-api-access-qf8g4\" (UniqueName: \"kubernetes.io/projected/e83c0e54-ebb9-4621-ab5a-f95010eb9158-kube-api-access-qf8g4\") pod \"e83c0e54-ebb9-4621-ab5a-f95010eb9158\" (UID: \"e83c0e54-ebb9-4621-ab5a-f95010eb9158\") "
Oct 31 19:15:17.822922 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.822822    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e83c0e54-ebb9-4621-ab5a-f95010eb9158-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83c0e54-ebb9-4621-ab5a-f95010eb9158-maas-api-service-ca\") pod \"e83c0e54-ebb9-4621-ab5a-f95010eb9158\" (UID: \"e83c0e54-ebb9-4621-ab5a-f95010eb9158\") "
Oct 31 19:15:17.823170 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.823150    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e83c0e54-ebb9-4621-ab5a-f95010eb9158-maas-api-service-ca" pod "e83c0e54-ebb9-4621-ab5a-f95010eb9158" (UID: "e83c0e54-ebb9-4621-ab5a-f95010eb9158"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:15:17.823297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.823281    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83c0e54-ebb9-4621-ab5a-f95010eb9158-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:17.824880 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.824855    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83c0e54-ebb9-4621-ab5a-f95010eb9158-kube-api-access-qf8g4" pod "e83c0e54-ebb9-4621-ab5a-f95010eb9158" (UID: "e83c0e54-ebb9-4621-ab5a-f95010eb9158"). InnerVolumeSpecName "kube-api-access-qf8g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:17.845325 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.845294    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"
Oct 31 19:15:17.924572 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.924466    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9aefadc-e01e-4c43-92ee-94f11332c501-kube-api-access-2kgwl\" (UniqueName: \"kubernetes.io/projected/a9aefadc-e01e-4c43-92ee-94f11332c501-kube-api-access-2kgwl\") pod \"a9aefadc-e01e-4c43-92ee-94f11332c501\" (UID: \"a9aefadc-e01e-4c43-92ee-94f11332c501\") "
Oct 31 19:15:17.924744 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.924565    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9aefadc-e01e-4c43-92ee-94f11332c501-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aefadc-e01e-4c43-92ee-94f11332c501-maas-api-service-ca\") pod \"a9aefadc-e01e-4c43-92ee-94f11332c501\" (UID: \"a9aefadc-e01e-4c43-92ee-94f11332c501\") "
Oct 31 19:15:17.924961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.924934    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9aefadc-e01e-4c43-92ee-94f11332c501-maas-api-service-ca" pod "a9aefadc-e01e-4c43-92ee-94f11332c501" (UID: "a9aefadc-e01e-4c43-92ee-94f11332c501"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:15:17.925099 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.925082    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9aefadc-e01e-4c43-92ee-94f11332c501-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:17.925174 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.925102    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf8g4\" (UniqueName: \"kubernetes.io/projected/e83c0e54-ebb9-4621-ab5a-f95010eb9158-kube-api-access-qf8g4\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:17.927400 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.927369    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9aefadc-e01e-4c43-92ee-94f11332c501-kube-api-access-2kgwl" pod "a9aefadc-e01e-4c43-92ee-94f11332c501" (UID: "a9aefadc-e01e-4c43-92ee-94f11332c501"). InnerVolumeSpecName "kube-api-access-2kgwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:17.987298 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:17.987276    4484 scope.go:122] "RemoveContainer" containerID="4d42ff8c98f6f79ae796e89ae55797902d6ce5afefdc9232b17a42126afdbc16"
Oct 31 19:15:18.026324 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.026297    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kgwl\" (UniqueName: \"kubernetes.io/projected/a9aefadc-e01e-4c43-92ee-94f11332c501-kube-api-access-2kgwl\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:18.361750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.361715    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv" event={"ID":"440ed2aa-0722-4e4a-9434-24a93962abd1","Type":"ContainerStarted","Data":"74be55043d39bd3e8d8daca5ea36a83bdfa1a3b03deb1f0972f13740bb9942e9"}
Oct 31 19:15:18.364984 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.364960    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq" event={"ID":"a9aefadc-e01e-4c43-92ee-94f11332c501","Type":"ContainerDied","Data":"2acbc0a19dcf68effe8d9b9c5bba77275e45ab1c0da36e43fe56e83ad5e90c4d"}
Oct 31 19:15:18.365126 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.364964    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"
Oct 31 19:15:18.365126 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.365007    4484 scope.go:122] "RemoveContainer" containerID="622fff241a24e9214385a5321f716c5c3eb555e06c510b38a649ce4d925868d9"
Oct 31 19:15:18.366433 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.366411    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6" event={"ID":"e83c0e54-ebb9-4621-ab5a-f95010eb9158","Type":"ContainerDied","Data":"1a0f0773f21425962f34009e0ce71a12e2b1b79bd68965d3bf4a270caa1deefb"}
Oct 31 19:15:18.366611 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.366414    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"
Oct 31 19:15:18.377002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.376983    4484 scope.go:122] "RemoveContainer" containerID="7706c623e607070b8faf1571be46e5dee6b347e8b9825ee3404fe9c5af1e058d"
Oct 31 19:15:18.478202 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.478158    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv" podStartSLOduration=18.478143166 podStartE2EDuration="18.478143166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:15:18.477356715 +0000 UTC m=+3264.623806583" watchObservedRunningTime="2026-10-31 19:15:18.478143166 +0000 UTC m=+3264.624593029"
Oct 31 19:15:18.499192 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.499167    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"
Oct 31 19:15:18.532523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.532493    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc2126d-aec4-4cff-983b-2c760030227c-kube-api-access-6ztbz\" (UniqueName: \"kubernetes.io/projected/3bc2126d-aec4-4cff-983b-2c760030227c-kube-api-access-6ztbz\") pod \"3bc2126d-aec4-4cff-983b-2c760030227c\" (UID: \"3bc2126d-aec4-4cff-983b-2c760030227c\") "
Oct 31 19:15:18.532664 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.532625    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc2126d-aec4-4cff-983b-2c760030227c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc2126d-aec4-4cff-983b-2c760030227c-maas-api-service-ca\") pod \"3bc2126d-aec4-4cff-983b-2c760030227c\" (UID: \"3bc2126d-aec4-4cff-983b-2c760030227c\") "
Oct 31 19:15:18.533021 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.532993    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc2126d-aec4-4cff-983b-2c760030227c-maas-api-service-ca" pod "3bc2126d-aec4-4cff-983b-2c760030227c" (UID: "3bc2126d-aec4-4cff-983b-2c760030227c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:15:18.533309 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.533235    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc2126d-aec4-4cff-983b-2c760030227c-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:18.534814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.534790    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc2126d-aec4-4cff-983b-2c760030227c-kube-api-access-6ztbz" pod "3bc2126d-aec4-4cff-983b-2c760030227c" (UID: "3bc2126d-aec4-4cff-983b-2c760030227c"). InnerVolumeSpecName "kube-api-access-6ztbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:18.592852 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.592817    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"]
Oct 31 19:15:18.597839 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.597815    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dbceb9-29891235-qgppq"]
Oct 31 19:15:18.634951 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.634889    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ztbz\" (UniqueName: \"kubernetes.io/projected/3bc2126d-aec4-4cff-983b-2c760030227c-kube-api-access-6ztbz\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:18.819885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.819850    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"]
Oct 31 19:15:18.824706 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:18.824670    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891235-t6pl6"]
Oct 31 19:15:19.374063 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.374030    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"
Oct 31 19:15:19.374623 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.374031    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8" event={"ID":"3bc2126d-aec4-4cff-983b-2c760030227c","Type":"ContainerDied","Data":"68e8be6edd63bc725de1b777e5989f094f3e9e284b38a055444685114cf513c2"}
Oct 31 19:15:19.374623 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.374230    4484 scope.go:122] "RemoveContainer" containerID="9a02a2ce30431b751480e2be74d988f8af3a2d7f92b97ed7577af5988a0bcf31"
Oct 31 19:15:19.378326 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.378294    4484 generic.go:356] "Generic (PLEG): container finished" podID="440ed2aa-0722-4e4a-9434-24a93962abd1" containerID="74be55043d39bd3e8d8daca5ea36a83bdfa1a3b03deb1f0972f13740bb9942e9" exitCode=60
Oct 31 19:15:19.378419 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.378323    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv" event={"ID":"440ed2aa-0722-4e4a-9434-24a93962abd1","Type":"ContainerDied","Data":"74be55043d39bd3e8d8daca5ea36a83bdfa1a3b03deb1f0972f13740bb9942e9"}
Oct 31 19:15:19.378705 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.378691    4484 scope.go:122] "RemoveContainer" containerID="74be55043d39bd3e8d8daca5ea36a83bdfa1a3b03deb1f0972f13740bb9942e9"
Oct 31 19:15:19.378973 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:19.378955    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv_odh-ai-gateway-infra(440ed2aa-0722-4e4a-9434-24a93962abd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1"
Oct 31 19:15:19.385457 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.385440    4484 scope.go:122] "RemoveContainer" containerID="4d42ff8c98f6f79ae796e89ae55797902d6ce5afefdc9232b17a42126afdbc16"
Oct 31 19:15:19.410657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.410630    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:19.489113 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.489084    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"]
Oct 31 19:15:19.659813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.659736    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"]
Oct 31 19:15:19.662852 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.662826    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-d9922f-29891235-7n5g8"]
Oct 31 19:15:19.996637 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.996607    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc2126d-aec4-4cff-983b-2c760030227c" path="/var/lib/kubelet/pods/3bc2126d-aec4-4cff-983b-2c760030227c/volumes"
Oct 31 19:15:19.997111 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.997091    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501" path="/var/lib/kubelet/pods/a9aefadc-e01e-4c43-92ee-94f11332c501/volumes"
Oct 31 19:15:19.997588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:19.997568    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" path="/var/lib/kubelet/pods/e83c0e54-ebb9-4621-ab5a-f95010eb9158/volumes"
Oct 31 19:15:20.553633 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:20.553606    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"
Oct 31 19:15:20.657553 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:20.657436    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440ed2aa-0722-4e4a-9434-24a93962abd1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/440ed2aa-0722-4e4a-9434-24a93962abd1-maas-api-service-ca\") pod \"440ed2aa-0722-4e4a-9434-24a93962abd1\" (UID: \"440ed2aa-0722-4e4a-9434-24a93962abd1\") "
Oct 31 19:15:20.657738 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:20.657629    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440ed2aa-0722-4e4a-9434-24a93962abd1-kube-api-access-qsvx7\" (UniqueName: \"kubernetes.io/projected/440ed2aa-0722-4e4a-9434-24a93962abd1-kube-api-access-qsvx7\") pod \"440ed2aa-0722-4e4a-9434-24a93962abd1\" (UID: \"440ed2aa-0722-4e4a-9434-24a93962abd1\") "
Oct 31 19:15:20.657808 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:20.657786    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440ed2aa-0722-4e4a-9434-24a93962abd1-maas-api-service-ca" pod "440ed2aa-0722-4e4a-9434-24a93962abd1" (UID: "440ed2aa-0722-4e4a-9434-24a93962abd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:15:20.658023 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:20.658004    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/440ed2aa-0722-4e4a-9434-24a93962abd1-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:20.659805 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:20.659775    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440ed2aa-0722-4e4a-9434-24a93962abd1-kube-api-access-qsvx7" pod "440ed2aa-0722-4e4a-9434-24a93962abd1" (UID: "440ed2aa-0722-4e4a-9434-24a93962abd1"). InnerVolumeSpecName "kube-api-access-qsvx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:15:20.758721 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:20.758687    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsvx7\" (UniqueName: \"kubernetes.io/projected/440ed2aa-0722-4e4a-9434-24a93962abd1-kube-api-access-qsvx7\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:15:21.396683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:21.396654    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"
Oct 31 19:15:21.396901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:21.396654    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv" event={"ID":"440ed2aa-0722-4e4a-9434-24a93962abd1","Type":"ContainerDied","Data":"40bf25b4b1f4a5e5cce6e408d1be7d133f877f423ab570a26b8a016b7cc4ca24"}
Oct 31 19:15:21.396901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:21.396793    4484 scope.go:122] "RemoveContainer" containerID="74be55043d39bd3e8d8daca5ea36a83bdfa1a3b03deb1f0972f13740bb9942e9"
Oct 31 19:15:21.561556 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:21.561522    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"]
Oct 31 19:15:21.567736 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:21.567708    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891235-bn7rv"]
Oct 31 19:15:22.001559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:22.001522    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1" path="/var/lib/kubelet/pods/440ed2aa-0722-4e4a-9434-24a93962abd1/volumes"
Oct 31 19:15:24.056506 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:24.056460    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:25.469444 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:25.469407    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:15:28.686668 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:28.686623    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:30.034036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:30.033995    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:33.343228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:33.343197    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:36.589972 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:36.589880    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:38.072836 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:38.072800    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:41.623822 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:41.623783    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:41.957322 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:41.957209    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:42.754280 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:42.754246    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:47.466714 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:47.466661    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:49.999534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:49.999504    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:50.942298 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:50.942262    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:52.150942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:52.150903    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:54.694145 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:54.694115    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:15:54.695844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:54.695825    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:15:55.469096 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:15:55.469061    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:15:56.835162 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:56.835122    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:15:58.816765 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:15:58.816727    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:01.584005 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:01.583954    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:03.308540 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:03.308507    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:03.745690 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:03.745662    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:16:06.264523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:06.264459    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:09.637742 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:09.637701    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:10.275361 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:10.275322    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:10.682201 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:10.682102    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:11.044900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:11.044868    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:14.584371 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:14.584336    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:14.757840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:14.757803    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:14.944080 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:14.943987    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:15.790250 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:15.790218    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"]
Oct 31 19:16:15.881774 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:15.881741    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:15.894712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:15.894683    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"]
Oct 31 19:16:16.162768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:16.162679    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:16.312273 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:16.312237    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:16.658104 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:16.658070    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:19.554358 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:19.554321    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:19.746911 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:19.746877    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:20.790632 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:20.790566    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v" podUID="a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4" containerName="payload-processing" containerID="cri-o://b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886" gracePeriod=25
Oct 31 19:16:20.895407 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:20.895363    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2" podUID="8788603e-7b27-4f88-aac2-e7ca790925d0" containerName="payload-pre-processing" containerID="cri-o://8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7" gracePeriod=25
Oct 31 19:16:20.952590 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:20.952550    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:21.067327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.067302    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"
Oct 31 19:16:21.168842 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.168811    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-plugins-config-volume\") pod \"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4\" (UID: \"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4\") "
Oct 31 19:16:21.168996 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.168855    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-kube-api-access-7xzcs\" (UniqueName: \"kubernetes.io/projected/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-kube-api-access-7xzcs\") pod \"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4\" (UID: \"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4\") "
Oct 31 19:16:21.169258 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.169223    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-plugins-config-volume" pod "a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4" (UID: "a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:16:21.171148 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.171123    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-kube-api-access-7xzcs" pod "a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4" (UID: "a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4"). InnerVolumeSpecName "kube-api-access-7xzcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:16:21.196553 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.196519    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"
Oct 31 19:16:21.269714 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.269684    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8788603e-7b27-4f88-aac2-e7ca790925d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8788603e-7b27-4f88-aac2-e7ca790925d0-plugins-config-volume\") pod \"8788603e-7b27-4f88-aac2-e7ca790925d0\" (UID: \"8788603e-7b27-4f88-aac2-e7ca790925d0\") "
Oct 31 19:16:21.269907 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.269802    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8788603e-7b27-4f88-aac2-e7ca790925d0-kube-api-access-2ht4p\" (UniqueName: \"kubernetes.io/projected/8788603e-7b27-4f88-aac2-e7ca790925d0-kube-api-access-2ht4p\") pod \"8788603e-7b27-4f88-aac2-e7ca790925d0\" (UID: \"8788603e-7b27-4f88-aac2-e7ca790925d0\") "
Oct 31 19:16:21.270072 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.270037    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8788603e-7b27-4f88-aac2-e7ca790925d0-plugins-config-volume" pod "8788603e-7b27-4f88-aac2-e7ca790925d0" (UID: "8788603e-7b27-4f88-aac2-e7ca790925d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:16:21.270321 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.270300    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:21.270508 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.270328    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8788603e-7b27-4f88-aac2-e7ca790925d0-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:21.270508 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.270341    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xzcs\" (UniqueName: \"kubernetes.io/projected/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4-kube-api-access-7xzcs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:21.272179 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.272144    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8788603e-7b27-4f88-aac2-e7ca790925d0-kube-api-access-2ht4p" pod "8788603e-7b27-4f88-aac2-e7ca790925d0" (UID: "8788603e-7b27-4f88-aac2-e7ca790925d0"). InnerVolumeSpecName "kube-api-access-2ht4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:16:21.371401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.371314    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ht4p\" (UniqueName: \"kubernetes.io/projected/8788603e-7b27-4f88-aac2-e7ca790925d0-kube-api-access-2ht4p\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:21.786153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.786118    4484 generic.go:356] "Generic (PLEG): container finished" podID="8788603e-7b27-4f88-aac2-e7ca790925d0" containerID="8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7" exitCode=0
Oct 31 19:16:21.786342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.786169    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"
Oct 31 19:16:21.786342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.786183    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2" event={"ID":"8788603e-7b27-4f88-aac2-e7ca790925d0","Type":"ContainerDied","Data":"8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7"}
Oct 31 19:16:21.786342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.786244    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2" event={"ID":"8788603e-7b27-4f88-aac2-e7ca790925d0","Type":"ContainerDied","Data":"41ba7c49b8c02c15505c11a3af9a7178fb725cac506b98b51d35dd6aa4ac336b"}
Oct 31 19:16:21.786342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.786277    4484 scope.go:122] "RemoveContainer" containerID="8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7"
Oct 31 19:16:21.788066 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.788035    4484 generic.go:356] "Generic (PLEG): container finished" podID="a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4" containerID="b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886" exitCode=0
Oct 31 19:16:21.788209 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.788087    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v" event={"ID":"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4","Type":"ContainerDied","Data":"b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886"}
Oct 31 19:16:21.788209 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.788118    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v" event={"ID":"a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4","Type":"ContainerDied","Data":"33dbd33e465777cdea2b03d79c040e88ff6fcc74b6260a1398452f5ed416b1ef"}
Oct 31 19:16:21.788209 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.788172    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"
Oct 31 19:16:21.798968 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.798759    4484 scope.go:122] "RemoveContainer" containerID="8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7"
Oct 31 19:16:21.799265 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:16:21.799057    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7\": container with ID starting with 8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7 not found: ID does not exist" containerID="8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7"
Oct 31 19:16:21.799265 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.799091    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7"} err="failed to get container status \"8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7\": rpc error: code = NotFound desc = could not find container \"8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7\": container with ID starting with 8838b247d4c9145360e45d4071f79f14b10190187c79c5967086925d851feda7 not found: ID does not exist"
Oct 31 19:16:21.799265 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.799111    4484 scope.go:122] "RemoveContainer" containerID="b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886"
Oct 31 19:16:21.817507 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.817487    4484 scope.go:122] "RemoveContainer" containerID="b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886"
Oct 31 19:16:21.817832 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:16:21.817812    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886\": container with ID starting with b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886 not found: ID does not exist" containerID="b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886"
Oct 31 19:16:21.817880 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.817842    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886"} err="failed to get container status \"b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886\": rpc error: code = NotFound desc = could not find container \"b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886\": container with ID starting with b87983a79960f99ae0f89881b7dd34b4de8e518f4c010c90c5fa082e15ad8886 not found: ID does not exist"
Oct 31 19:16:21.958942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.958912    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"]
Oct 31 19:16:21.962753 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.962722    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-d9922f-6c6bbf9d99-pd8d2"]
Oct 31 19:16:21.993908 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:21.993881    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8788603e-7b27-4f88-aac2-e7ca790925d0" path="/var/lib/kubelet/pods/8788603e-7b27-4f88-aac2-e7ca790925d0/volumes"
Oct 31 19:16:22.075721 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:22.075637    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"]
Oct 31 19:16:22.079805 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:22.079775    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-d9922f-68bcf4bcb9-5cp9v"]
Oct 31 19:16:22.724213 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:22.724169    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:23.995954 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:23.995927    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4" path="/var/lib/kubelet/pods/a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4/volumes"
Oct 31 19:16:25.468735 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:16:25.468696    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:16:25.774324 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:25.774272    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:30.650016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:30.649984    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:32.227991 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:32.227948    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:35.319185 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:35.319158    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:38.796505 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:38.796449    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:39.390497 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:39.390447    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:39.591689 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:39.591656    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:39.748193 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:39.748144    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:40.070598 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:40.070502    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:43.704281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:43.704227    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:43.872631 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:43.872590    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:44.037023 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:44.036996    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:44.977110 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:44.977070    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:46.908913 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:46.908876    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:47.138782 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.138748    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:47.315041 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315009    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"]
Oct 31 19:16:47.315689 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315675    4484 cpu_manager.go:414] "containerMap: removing container" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1" containerName="cleanup"
Oct 31 19:16:47.315750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315694    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1" containerName="cleanup"
Oct 31 19:16:47.315750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315710    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501" containerName="cleanup"
Oct 31 19:16:47.315750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315716    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501" containerName="cleanup"
Oct 31 19:16:47.315750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315726    4484 cpu_manager.go:414] "containerMap: removing container" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" containerName="cleanup"
Oct 31 19:16:47.315750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315731    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" containerName="cleanup"
Oct 31 19:16:47.315750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315738    4484 cpu_manager.go:414] "containerMap: removing container" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" containerName="cleanup"
Oct 31 19:16:47.315750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315743    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" containerName="cleanup"
Oct 31 19:16:47.315750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315751    4484 cpu_manager.go:414] "containerMap: removing container" podUID="3bc2126d-aec4-4cff-983b-2c760030227c" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315756    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc2126d-aec4-4cff-983b-2c760030227c" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315762    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315766    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315774    4484 cpu_manager.go:414] "containerMap: removing container" podUID="8788603e-7b27-4f88-aac2-e7ca790925d0" containerName="payload-pre-processing"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315778    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8788603e-7b27-4f88-aac2-e7ca790925d0" containerName="payload-pre-processing"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315798    4484 cpu_manager.go:414] "containerMap: removing container" podUID="3bc2126d-aec4-4cff-983b-2c760030227c" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315802    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc2126d-aec4-4cff-983b-2c760030227c" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315808    4484 cpu_manager.go:414] "containerMap: removing container" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315812    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315839    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4" containerName="payload-processing"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315845    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4" containerName="payload-processing"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315934    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315942    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc2126d-aec4-4cff-983b-2c760030227c" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315948    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315954    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315959    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ec962b-4a6f-40f7-b07f-2c2e7543e8a4" containerName="payload-processing"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315965    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315972    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315977    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501" containerName="cleanup"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315982    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="8788603e-7b27-4f88-aac2-e7ca790925d0" containerName="payload-pre-processing"
Oct 31 19:16:47.316002 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.315987    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc2126d-aec4-4cff-983b-2c760030227c" containerName="cleanup"
Oct 31 19:16:47.319345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.319326    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"
Oct 31 19:16:47.376989 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.376961    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"]
Oct 31 19:16:47.385716 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.385682    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpzb2\" (UniqueName: \"kubernetes.io/projected/85defdb9-17ce-48a9-8785-0171ab21881a-kube-api-access-fpzb2\") pod \"payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99\" (UID: \"85defdb9-17ce-48a9-8785-0171ab21881a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"
Oct 31 19:16:47.385875 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.385750    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85defdb9-17ce-48a9-8785-0171ab21881a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99\" (UID: \"85defdb9-17ce-48a9-8785-0171ab21881a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"
Oct 31 19:16:47.423685 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.423657    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-75c484\""
Oct 31 19:16:47.424057 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.424042    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-75c484-dockercfg-2wzw7\""
Oct 31 19:16:47.479413 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.479365    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"]
Oct 31 19:16:47.480586 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.480546    4484 cpu_manager.go:414] "containerMap: removing container" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1" containerName="cleanup"
Oct 31 19:16:47.480840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.480812    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1" containerName="cleanup"
Oct 31 19:16:47.480942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.480859    4484 cpu_manager.go:414] "containerMap: removing container" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" containerName="cleanup"
Oct 31 19:16:47.480942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.480867    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" containerName="cleanup"
Oct 31 19:16:47.480942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.480883    4484 cpu_manager.go:414] "containerMap: removing container" podUID="3bc2126d-aec4-4cff-983b-2c760030227c" containerName="cleanup"
Oct 31 19:16:47.480942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.480891    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc2126d-aec4-4cff-983b-2c760030227c" containerName="cleanup"
Oct 31 19:16:47.480942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.480901    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501" containerName="cleanup"
Oct 31 19:16:47.480942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.480908    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501" containerName="cleanup"
Oct 31 19:16:47.481320 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.481112    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e83c0e54-ebb9-4621-ab5a-f95010eb9158" containerName="cleanup"
Oct 31 19:16:47.481320 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.481128    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc2126d-aec4-4cff-983b-2c760030227c" containerName="cleanup"
Oct 31 19:16:47.481320 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.481143    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9aefadc-e01e-4c43-92ee-94f11332c501" containerName="cleanup"
Oct 31 19:16:47.481320 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.481155    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="440ed2aa-0722-4e4a-9434-24a93962abd1" containerName="cleanup"
Oct 31 19:16:47.484950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.484931    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"
Oct 31 19:16:47.487045 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.487022    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpzb2\" (UniqueName: \"kubernetes.io/projected/85defdb9-17ce-48a9-8785-0171ab21881a-kube-api-access-fpzb2\") pod \"payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99\" (UID: \"85defdb9-17ce-48a9-8785-0171ab21881a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"
Oct 31 19:16:47.487158 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.487076    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85defdb9-17ce-48a9-8785-0171ab21881a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99\" (UID: \"85defdb9-17ce-48a9-8785-0171ab21881a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"
Oct 31 19:16:47.487848 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.487829    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85defdb9-17ce-48a9-8785-0171ab21881a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99\" (UID: \"85defdb9-17ce-48a9-8785-0171ab21881a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"
Oct 31 19:16:47.529401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.529370    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"]
Oct 31 19:16:47.544579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.544546    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpzb2\" (UniqueName: \"kubernetes.io/projected/85defdb9-17ce-48a9-8785-0171ab21881a-kube-api-access-fpzb2\") pod \"payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99\" (UID: \"85defdb9-17ce-48a9-8785-0171ab21881a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"
Oct 31 19:16:47.588926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.588660    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtpkb\" (UniqueName: \"kubernetes.io/projected/13a2c839-c270-412a-a636-17ee576a7c7f-kube-api-access-wtpkb\") pod \"payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r\" (UID: \"13a2c839-c270-412a-a636-17ee576a7c7f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"
Oct 31 19:16:47.588926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.588748    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a2c839-c270-412a-a636-17ee576a7c7f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r\" (UID: \"13a2c839-c270-412a-a636-17ee576a7c7f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"
Oct 31 19:16:47.632547 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.632516    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"
Oct 31 19:16:47.689977 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.689935    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtpkb\" (UniqueName: \"kubernetes.io/projected/13a2c839-c270-412a-a636-17ee576a7c7f-kube-api-access-wtpkb\") pod \"payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r\" (UID: \"13a2c839-c270-412a-a636-17ee576a7c7f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"
Oct 31 19:16:47.690129 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.690026    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a2c839-c270-412a-a636-17ee576a7c7f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r\" (UID: \"13a2c839-c270-412a-a636-17ee576a7c7f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"
Oct 31 19:16:47.691028 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.690997    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a2c839-c270-412a-a636-17ee576a7c7f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r\" (UID: \"13a2c839-c270-412a-a636-17ee576a7c7f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"
Oct 31 19:16:47.748257 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.748224    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtpkb\" (UniqueName: \"kubernetes.io/projected/13a2c839-c270-412a-a636-17ee576a7c7f-kube-api-access-wtpkb\") pod \"payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r\" (UID: \"13a2c839-c270-412a-a636-17ee576a7c7f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"
Oct 31 19:16:47.800202 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:47.800173    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"
Oct 31 19:16:48.005676 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:48.005650    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"]
Oct 31 19:16:48.006989 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:16:48.006967    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85defdb9_17ce_48a9_8785_0171ab21881a.slice/crio-c6eda40e03ee5473bda08b1159d2147dd2088b1c2cff6e0c2144a99cd26be695 WatchSource:0}: Error finding container c6eda40e03ee5473bda08b1159d2147dd2088b1c2cff6e0c2144a99cd26be695: Status 404 returned error can't find the container with id c6eda40e03ee5473bda08b1159d2147dd2088b1c2cff6e0c2144a99cd26be695
Oct 31 19:16:48.175157 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:48.175130    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"]
Oct 31 19:16:48.177818 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:16:48.177787    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a2c839_c270_412a_a636_17ee576a7c7f.slice/crio-7deffbb901ed4af87db8c7250063789c9da8f31637327fbaa73f877f0fc282fc WatchSource:0}: Error finding container 7deffbb901ed4af87db8c7250063789c9da8f31637327fbaa73f877f0fc282fc: Status 404 returned error can't find the container with id 7deffbb901ed4af87db8c7250063789c9da8f31637327fbaa73f877f0fc282fc
Oct 31 19:16:48.509816 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:48.509783    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:48.958207 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:48.958113    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99" event={"ID":"85defdb9-17ce-48a9-8785-0171ab21881a","Type":"ContainerStarted","Data":"8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a"}
Oct 31 19:16:48.958207 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:48.958176    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99" event={"ID":"85defdb9-17ce-48a9-8785-0171ab21881a","Type":"ContainerStarted","Data":"c6eda40e03ee5473bda08b1159d2147dd2088b1c2cff6e0c2144a99cd26be695"}
Oct 31 19:16:48.958207 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:48.958203    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"
Oct 31 19:16:48.959836 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:48.959804    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r" event={"ID":"13a2c839-c270-412a-a636-17ee576a7c7f","Type":"ContainerStarted","Data":"809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb"}
Oct 31 19:16:48.959979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:48.959851    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"
Oct 31 19:16:48.959979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:48.959867    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r" event={"ID":"13a2c839-c270-412a-a636-17ee576a7c7f","Type":"ContainerStarted","Data":"7deffbb901ed4af87db8c7250063789c9da8f31637327fbaa73f877f0fc282fc"}
Oct 31 19:16:49.076967 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.076907    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99" podStartSLOduration=2.076891348 podStartE2EDuration="2.076891348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:49.075524389 +0000 UTC m=+3355.221974252" watchObservedRunningTime="2026-10-31 19:16:49.076891348 +0000 UTC m=+3355.223341212"
Oct 31 19:16:49.194092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.194033    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r" podStartSLOduration=2.194017847 podStartE2EDuration="2.194017847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:16:49.191160714 +0000 UTC m=+3355.337610578" watchObservedRunningTime="2026-10-31 19:16:49.194017847 +0000 UTC m=+3355.340467714"
Oct 31 19:16:49.398363 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.398319    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"]
Oct 31 19:16:49.398630 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.398603    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg" podUID="05de422c-b712-4e93-af92-d42732e69015" containerName="maas-api" containerID="cri-o://28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd" gracePeriod=30
Oct 31 19:16:49.585323 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.585256    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:49.697663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.697635    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:16:49.814454 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.814416    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05de422c-b712-4e93-af92-d42732e69015-kube-api-access-k76t4\" (UniqueName: \"kubernetes.io/projected/05de422c-b712-4e93-af92-d42732e69015-kube-api-access-k76t4\") pod \"05de422c-b712-4e93-af92-d42732e69015\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") "
Oct 31 19:16:49.814688 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.814467    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05de422c-b712-4e93-af92-d42732e69015-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05de422c-b712-4e93-af92-d42732e69015-openshift-service-ca\") pod \"05de422c-b712-4e93-af92-d42732e69015\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") "
Oct 31 19:16:49.814766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.814702    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-metrics-certs\") pod \"05de422c-b712-4e93-af92-d42732e69015\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") "
Oct 31 19:16:49.814766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.814752    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-maas-api-tls\") pod \"05de422c-b712-4e93-af92-d42732e69015\" (UID: \"05de422c-b712-4e93-af92-d42732e69015\") "
Oct 31 19:16:49.815055 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.815028    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05de422c-b712-4e93-af92-d42732e69015-openshift-service-ca" pod "05de422c-b712-4e93-af92-d42732e69015" (UID: "05de422c-b712-4e93-af92-d42732e69015"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:16:49.815263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.815243    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05de422c-b712-4e93-af92-d42732e69015-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:49.818012 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.817972    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-metrics-certs" pod "05de422c-b712-4e93-af92-d42732e69015" (UID: "05de422c-b712-4e93-af92-d42732e69015"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:16:49.818271 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.818248    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-maas-api-tls" pod "05de422c-b712-4e93-af92-d42732e69015" (UID: "05de422c-b712-4e93-af92-d42732e69015"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:16:49.818637 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.818599    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05de422c-b712-4e93-af92-d42732e69015-kube-api-access-k76t4" pod "05de422c-b712-4e93-af92-d42732e69015" (UID: "05de422c-b712-4e93-af92-d42732e69015"). InnerVolumeSpecName "kube-api-access-k76t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:16:49.915970 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.915937    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:49.915970 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.915967    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05de422c-b712-4e93-af92-d42732e69015-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:49.915970 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.915977    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k76t4\" (UniqueName: \"kubernetes.io/projected/05de422c-b712-4e93-af92-d42732e69015-kube-api-access-k76t4\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:16:49.967972 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.967886    4484 generic.go:356] "Generic (PLEG): container finished" podID="05de422c-b712-4e93-af92-d42732e69015" containerID="28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd" exitCode=0
Oct 31 19:16:49.967972 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.967945    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"
Oct 31 19:16:49.968180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.967996    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg" event={"ID":"05de422c-b712-4e93-af92-d42732e69015","Type":"ContainerDied","Data":"28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd"}
Oct 31 19:16:49.968180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.968108    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg" event={"ID":"05de422c-b712-4e93-af92-d42732e69015","Type":"ContainerDied","Data":"58e6b55495c95638aff9563225ebfd4e6ac0f8d6383e5b83ea606fcf0ce0346d"}
Oct 31 19:16:49.968180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.968139    4484 scope.go:122] "RemoveContainer" containerID="28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd"
Oct 31 19:16:49.983702 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.983675    4484 scope.go:122] "RemoveContainer" containerID="28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd"
Oct 31 19:16:49.984314 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:16:49.984282    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd\": container with ID starting with 28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd not found: ID does not exist" containerID="28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd"
Oct 31 19:16:49.984555 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:49.984523    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd"} err="failed to get container status \"28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd\": rpc error: code = NotFound desc = could not find container \"28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd\": container with ID starting with 28708a791b858a2e946cbe034d5283069387ffddfb5422ec4e1566f463aa42bd not found: ID does not exist"
Oct 31 19:16:50.137510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:50.137465    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"]
Oct 31 19:16:50.141182 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:50.141153    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-d9922f-5dd95fdf86-lmdqg"]
Oct 31 19:16:51.284527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:51.284465    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:51.994298 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:51.994265    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05de422c-b712-4e93-af92-d42732e69015" path="/var/lib/kubelet/pods/05de422c-b712-4e93-af92-d42732e69015/volumes"
Oct 31 19:16:55.051841 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:55.051803    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:55.221928 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:55.221890    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:55.470224 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:16:55.470188    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:16:56.060734 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:56.060700    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:56.624372 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:56.624340    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:57.142408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:57.142373    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:57.767138 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:57.767100    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:58.566811 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:58.566750    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:58.784027 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:58.783994    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:59.167949 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:59.167905    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:16:59.972260 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:59.972233    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"
Oct 31 19:16:59.972781 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:16:59.972313    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"
Oct 31 19:17:00.612977 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:00.612942    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:00.825363 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:00.825311    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:00.989709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:00.989680    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:01.278521 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:01.278409    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:03.676769 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:03.676733    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:05.705466 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:05.705383    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:05.872567 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:05.872530    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:06.026355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:06.026314    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:10.633774 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:10.633735    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:14.536965 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:14.536924    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:15.276443 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:15.276399    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:19.950813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:19.950772    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:20.327249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:20.327221    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:17:21.324108 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:21.324009    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:24.624656 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:24.624618    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:24.814827 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:24.814785    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:24.978174 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:24.978057    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:25.469211 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:17:25.469173    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:17:28.951495 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:28.951442    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:29.126363 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:29.126327    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:29.531138 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:29.531099    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:34.222269 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:34.222223    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:38.918332 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:38.918291    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:43.704057 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:43.704017    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:48.398327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:48.398280    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:53.185916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:53.185882    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:54.299731 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:54.299690    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:55.471271 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:17:55.471233    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:17:57.979220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:57.979182    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:58.173698 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:58.173662    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:17:59.534737 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:17:59.534704    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:02.996928 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:02.996888    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:04.012800 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:04.012758    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:07.855832 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:07.855793    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:12.351728 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:12.351689    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:12.707383 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:12.707275    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:13.354709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:13.354670    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:13.530092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:13.530052    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:13.787014 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:13.786970    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:13.998406 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:13.998370    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:17.543065 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:17.543026    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:19.097414 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:19.097373    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:19.755717 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:19.755673    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:18:22.615191 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:22.615147    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:25.470387 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:18:25.470348    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:18:27.523538 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:27.523496    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:32.497176 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:32.497135    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:37.286332 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:37.286240    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:38.077917 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:38.077877    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:39.554392 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:39.554359    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:40.443078 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.443039    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"]
Oct 31 19:18:40.443957 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.443929    4484 cpu_manager.go:414] "containerMap: removing container" podUID="05de422c-b712-4e93-af92-d42732e69015" containerName="maas-api"
Oct 31 19:18:40.444097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.443969    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05de422c-b712-4e93-af92-d42732e69015" containerName="maas-api"
Oct 31 19:18:40.444218 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.444203    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="05de422c-b712-4e93-af92-d42732e69015" containerName="maas-api"
Oct 31 19:18:40.448354 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.448332    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.515033 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.515000    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49c681c-161a-47bd-accc-33a7c1735717-istiod-ca-cert\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.515246 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.515121    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-credential-socket\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.515246 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.515223    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st9gw\" (UniqueName: \"kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-kube-api-access-st9gw\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.515377 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.515317    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-socket\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.515430 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.515376    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-envoy\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.515535 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.515514    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49c681c-161a-47bd-accc-33a7c1735717-istio-podinfo\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.515599 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.515583    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-data\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.515725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.515621    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-istio-token\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.515725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.515681    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-certs\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.552927 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.552888    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-1aefcd-openshift-default-dockercfg-xjbzs\""
Oct 31 19:18:40.558208 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.558175    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"]
Oct 31 19:18:40.582796 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.582754    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:40.616856 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.616815    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49c681c-161a-47bd-accc-33a7c1735717-istio-podinfo\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617054 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.616898    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-data\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617054 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.616917    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-istio-token\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617054 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.616944    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-certs\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617239 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.617091    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49c681c-161a-47bd-accc-33a7c1735717-istiod-ca-cert\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617239 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.617173    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-credential-socket\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617370 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.617278    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st9gw\" (UniqueName: \"kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-kube-api-access-st9gw\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617370 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.617353    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-socket\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617560 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.617411    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-envoy\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617560 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.617422    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-data\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617560 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.617451    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-certs\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617560 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.617519    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-credential-socket\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617858 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.617837    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-socket\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.617921 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.617843    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49c681c-161a-47bd-accc-33a7c1735717-istiod-ca-cert\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.619807 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.619781    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-envoy\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.619919 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.619865    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49c681c-161a-47bd-accc-33a7c1735717-istio-podinfo\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.682355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.682320    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-istio-token\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.683130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.683103    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st9gw\" (UniqueName: \"kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-kube-api-access-st9gw\") pod \"e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") " pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.763052 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.763019    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:40.839115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:40.838986    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:41.200015 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:41.199972    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"]
Oct 31 19:18:41.201262 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:18:41.201226    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49c681c_161a_47bd_accc_33a7c1735717.slice/crio-cb0b4efe7dac3b180e9a7aff2743eede0044f0da8c2e85ffa7653cd04f34ea3d WatchSource:0}: Error finding container cb0b4efe7dac3b180e9a7aff2743eede0044f0da8c2e85ffa7653cd04f34ea3d: Status 404 returned error can't find the container with id cb0b4efe7dac3b180e9a7aff2743eede0044f0da8c2e85ffa7653cd04f34ea3d
Oct 31 19:18:41.203915 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:41.203884    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:18:41.203992 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:41.203983    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:18:41.204064 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:41.204012    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:18:41.673334 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:41.673300    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg" event={"ID":"a49c681c-161a-47bd-accc-33a7c1735717","Type":"ContainerStarted","Data":"59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129"}
Oct 31 19:18:41.673766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:41.673361    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg" event={"ID":"a49c681c-161a-47bd-accc-33a7c1735717","Type":"ContainerStarted","Data":"cb0b4efe7dac3b180e9a7aff2743eede0044f0da8c2e85ffa7653cd04f34ea3d"}
Oct 31 19:18:41.764166 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:41.764133    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:41.804662 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:41.804601    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg" podStartSLOduration=1.8045866959999999 podStartE2EDuration="1.804586696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:41.801601216 +0000 UTC m=+3467.948051084" watchObservedRunningTime="2026-10-31 19:18:41.804586696 +0000 UTC m=+3467.951036632"
Oct 31 19:18:42.359313 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:42.359276    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:42.786960 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:42.786920    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:18:42.787368 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:42.786974    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg" podUID="a49c681c-161a-47bd-accc-33a7c1735717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 19:18:43.781500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:43.777940    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:18:43.781500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:43.778019    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg" podUID="a49c681c-161a-47bd-accc-33a7c1735717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 19:18:44.783502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:44.769968    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:18:44.783502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:44.770048    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg" podUID="a49c681c-161a-47bd-accc-33a7c1735717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 19:18:45.236066 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:45.235961    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:45.629645 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:45.629555    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:45.777629 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:45.777588    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:18:45.777933 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:45.777903    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg" podUID="a49c681c-161a-47bd-accc-33a7c1735717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 19:18:46.785502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:46.780598    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:18:46.785502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:46.780666    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg" podUID="a49c681c-161a-47bd-accc-33a7c1735717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 19:18:47.763925 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:47.763874    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:18:47.764150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:47.763942    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg" podUID="a49c681c-161a-47bd-accc-33a7c1735717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 19:18:47.926203 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:47.926170    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:48.768401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:48.768367    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:48.768830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:48.768813    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:48.769641 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:48.769622    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:18:52.055846 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:52.055808    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:52.326343 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:52.326266    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:53.060296 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:53.060258    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:53.264530 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:53.264500    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:54.627225 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:54.627191    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:55.268158 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:55.268121    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:55.469948 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:18:55.469912    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:18:55.484766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:55.484735    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:57.667044 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:57.667003    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:58.005790 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:58.005756    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:59.164746 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:59.164714    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:18:59.866569 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:18:59.866531    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:00.082851 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:00.082816    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:00.303843 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:00.303812    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:03.347194 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:03.347135    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:04.337851 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:04.337808    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:04.886872 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:04.886837    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:05.125314 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:05.125174    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:05.855415 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:05.855382    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:19:06.557554 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:06.557509    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:06.788815 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:06.788780    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:08.697057 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:08.697022    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:11.556583 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:11.556546    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:11.836038 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:11.835940    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:13.890036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:13.889994    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:18.858920 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:18.858870    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:23.603354 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:23.603319    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"]
Oct 31 19:19:23.674018 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:23.673986    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:23.956542 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:23.956443    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:25.469063 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:19:25.469030    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:19:27.144089 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:27.144022    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:28.471612 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:28.471577    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:28.801452 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:28.801413    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:29.079974 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:29.079887    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:29.599720 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:29.599686    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:33.173722 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:33.173684    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:33.362033 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:33.361990    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:33.543990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:33.543955    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:33.805811 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:33.805718    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:36.420799 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:36.420769    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:36.593210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:36.593163    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:36.822765 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:36.822732    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"]
Oct 31 19:19:36.827266 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:36.827243    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"
Oct 31 19:19:36.881713 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:36.881682    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-3744b0\""
Oct 31 19:19:36.912158 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:36.912111    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1496d6eb-f023-4dfe-8dfc-7b8667661896-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g\" (UID: \"1496d6eb-f023-4dfe-8dfc-7b8667661896\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"
Oct 31 19:19:36.912389 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:36.912271    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rpj\" (UniqueName: \"kubernetes.io/projected/1496d6eb-f023-4dfe-8dfc-7b8667661896-kube-api-access-p5rpj\") pod \"payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g\" (UID: \"1496d6eb-f023-4dfe-8dfc-7b8667661896\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"
Oct 31 19:19:36.932539 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:36.932509    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-3744b0-dockercfg-kmk6s\""
Oct 31 19:19:36.937892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:36.937858    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"]
Oct 31 19:19:36.986116 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:36.986084    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"]
Oct 31 19:19:36.990544 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:36.990526    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"
Oct 31 19:19:37.013697 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.013657    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1496d6eb-f023-4dfe-8dfc-7b8667661896-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g\" (UID: \"1496d6eb-f023-4dfe-8dfc-7b8667661896\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"
Oct 31 19:19:37.013861 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.013770    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rpj\" (UniqueName: \"kubernetes.io/projected/1496d6eb-f023-4dfe-8dfc-7b8667661896-kube-api-access-p5rpj\") pod \"payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g\" (UID: \"1496d6eb-f023-4dfe-8dfc-7b8667661896\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"
Oct 31 19:19:37.014364 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.014345    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1496d6eb-f023-4dfe-8dfc-7b8667661896-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g\" (UID: \"1496d6eb-f023-4dfe-8dfc-7b8667661896\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"
Oct 31 19:19:37.025627 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.025594    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"]
Oct 31 19:19:37.072516 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.072458    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rpj\" (UniqueName: \"kubernetes.io/projected/1496d6eb-f023-4dfe-8dfc-7b8667661896-kube-api-access-p5rpj\") pod \"payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g\" (UID: \"1496d6eb-f023-4dfe-8dfc-7b8667661896\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"
Oct 31 19:19:37.115070 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.114986    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c86b6\" (UniqueName: \"kubernetes.io/projected/108076ec-b5aa-4b14-bed6-70e3714f2a1e-kube-api-access-c86b6\") pod \"payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k\" (UID: \"108076ec-b5aa-4b14-bed6-70e3714f2a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"
Oct 31 19:19:37.115292 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.115274    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108076ec-b5aa-4b14-bed6-70e3714f2a1e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k\" (UID: \"108076ec-b5aa-4b14-bed6-70e3714f2a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"
Oct 31 19:19:37.141360 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.141330    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"
Oct 31 19:19:37.220804 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.216766    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c86b6\" (UniqueName: \"kubernetes.io/projected/108076ec-b5aa-4b14-bed6-70e3714f2a1e-kube-api-access-c86b6\") pod \"payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k\" (UID: \"108076ec-b5aa-4b14-bed6-70e3714f2a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"
Oct 31 19:19:37.220804 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.216997    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108076ec-b5aa-4b14-bed6-70e3714f2a1e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k\" (UID: \"108076ec-b5aa-4b14-bed6-70e3714f2a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"
Oct 31 19:19:37.220804 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.217987    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108076ec-b5aa-4b14-bed6-70e3714f2a1e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k\" (UID: \"108076ec-b5aa-4b14-bed6-70e3714f2a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"
Oct 31 19:19:37.278223 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.278190    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c86b6\" (UniqueName: \"kubernetes.io/projected/108076ec-b5aa-4b14-bed6-70e3714f2a1e-kube-api-access-c86b6\") pod \"payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k\" (UID: \"108076ec-b5aa-4b14-bed6-70e3714f2a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"
Oct 31 19:19:37.304618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.304585    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"
Oct 31 19:19:37.528537 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.528515    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"]
Oct 31 19:19:37.530111 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:19:37.530087    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1496d6eb_f023_4dfe_8dfc_7b8667661896.slice/crio-6398da713d69886d3165a2279cf4d4ca5567644341f7b9b874e71fac28c7e9b8 WatchSource:0}: Error finding container 6398da713d69886d3165a2279cf4d4ca5567644341f7b9b874e71fac28c7e9b8: Status 404 returned error can't find the container with id 6398da713d69886d3165a2279cf4d4ca5567644341f7b9b874e71fac28c7e9b8
Oct 31 19:19:37.686606 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:37.686576    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"]
Oct 31 19:19:37.686742 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:19:37.686649    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108076ec_b5aa_4b14_bed6_70e3714f2a1e.slice/crio-392f86cf38e9b2f0d80a1b9d23a770237e1e381cc76c4463bde0223411835daf WatchSource:0}: Error finding container 392f86cf38e9b2f0d80a1b9d23a770237e1e381cc76c4463bde0223411835daf: Status 404 returned error can't find the container with id 392f86cf38e9b2f0d80a1b9d23a770237e1e381cc76c4463bde0223411835daf
Oct 31 19:19:38.060576 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:38.060536    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g" event={"ID":"1496d6eb-f023-4dfe-8dfc-7b8667661896","Type":"ContainerStarted","Data":"49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d"}
Oct 31 19:19:38.060784 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:38.060600    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g" event={"ID":"1496d6eb-f023-4dfe-8dfc-7b8667661896","Type":"ContainerStarted","Data":"6398da713d69886d3165a2279cf4d4ca5567644341f7b9b874e71fac28c7e9b8"}
Oct 31 19:19:38.060784 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:38.060623    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"
Oct 31 19:19:38.062101 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:38.062074    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k" event={"ID":"108076ec-b5aa-4b14-bed6-70e3714f2a1e","Type":"ContainerStarted","Data":"cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f"}
Oct 31 19:19:38.062252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:38.062123    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k" event={"ID":"108076ec-b5aa-4b14-bed6-70e3714f2a1e","Type":"ContainerStarted","Data":"392f86cf38e9b2f0d80a1b9d23a770237e1e381cc76c4463bde0223411835daf"}
Oct 31 19:19:38.062252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:38.062153    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"
Oct 31 19:19:38.182028 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:38.181979    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g" podStartSLOduration=2.181965853 podStartE2EDuration="2.181965853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:38.177858908 +0000 UTC m=+3524.324308771" watchObservedRunningTime="2026-10-31 19:19:38.181965853 +0000 UTC m=+3524.328415717"
Oct 31 19:19:38.298969 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:38.298855    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k" podStartSLOduration=2.298834755 podStartE2EDuration="2.298834755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:38.296611315 +0000 UTC m=+3524.443061180" watchObservedRunningTime="2026-10-31 19:19:38.298834755 +0000 UTC m=+3524.445284618"
Oct 31 19:19:38.625113 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:38.625068    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z" podUID="f01e3592-abbe-44cb-bf71-5d3e039d4526" containerName="main" containerID="cri-o://4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20" gracePeriod=45
Oct 31 19:19:38.902955 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:38.902924    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:19:39.030725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.030679    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:39.041569 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.041537    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-tmp-dir\") pod \"f01e3592-abbe-44cb-bf71-5d3e039d4526\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") "
Oct 31 19:19:39.041807 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.041621    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f01e3592-abbe-44cb-bf71-5d3e039d4526-tls-certs\" (UniqueName: \"kubernetes.io/secret/f01e3592-abbe-44cb-bf71-5d3e039d4526-tls-certs\") pod \"f01e3592-abbe-44cb-bf71-5d3e039d4526\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") "
Oct 31 19:19:39.041901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.041837    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-home\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-home\") pod \"f01e3592-abbe-44cb-bf71-5d3e039d4526\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") "
Oct 31 19:19:39.041965 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.041916    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-model-cache\") pod \"f01e3592-abbe-44cb-bf71-5d3e039d4526\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") "
Oct 31 19:19:39.042020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.041963    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01e3592-abbe-44cb-bf71-5d3e039d4526-kube-api-access-htnj2\" (UniqueName: \"kubernetes.io/projected/f01e3592-abbe-44cb-bf71-5d3e039d4526-kube-api-access-htnj2\") pod \"f01e3592-abbe-44cb-bf71-5d3e039d4526\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") "
Oct 31 19:19:39.042128 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.042027    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-dshm\") pod \"f01e3592-abbe-44cb-bf71-5d3e039d4526\" (UID: \"f01e3592-abbe-44cb-bf71-5d3e039d4526\") "
Oct 31 19:19:39.042128 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.042029    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-tmp-dir" pod "f01e3592-abbe-44cb-bf71-5d3e039d4526" (UID: "f01e3592-abbe-44cb-bf71-5d3e039d4526"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:19:39.049527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.042301    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-home" pod "f01e3592-abbe-44cb-bf71-5d3e039d4526" (UID: "f01e3592-abbe-44cb-bf71-5d3e039d4526"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:19:39.049527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.042617    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:19:39.049527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.042644    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:19:39.049527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.045444    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-dshm" pod "f01e3592-abbe-44cb-bf71-5d3e039d4526" (UID: "f01e3592-abbe-44cb-bf71-5d3e039d4526"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:19:39.049527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.047678    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-model-cache" pod "f01e3592-abbe-44cb-bf71-5d3e039d4526" (UID: "f01e3592-abbe-44cb-bf71-5d3e039d4526"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:19:39.049527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.048294    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01e3592-abbe-44cb-bf71-5d3e039d4526-tls-certs" pod "f01e3592-abbe-44cb-bf71-5d3e039d4526" (UID: "f01e3592-abbe-44cb-bf71-5d3e039d4526"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:19:39.049527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.048454    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01e3592-abbe-44cb-bf71-5d3e039d4526-kube-api-access-htnj2" pod "f01e3592-abbe-44cb-bf71-5d3e039d4526" (UID: "f01e3592-abbe-44cb-bf71-5d3e039d4526"). InnerVolumeSpecName "kube-api-access-htnj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:19:39.069678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.069651    4484 generic.go:356] "Generic (PLEG): container finished" podID="f01e3592-abbe-44cb-bf71-5d3e039d4526" containerID="4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20" exitCode=0
Oct 31 19:19:39.069820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.069697    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"
Oct 31 19:19:39.069820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.069731    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z" event={"ID":"f01e3592-abbe-44cb-bf71-5d3e039d4526","Type":"ContainerDied","Data":"4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20"}
Oct 31 19:19:39.069820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.069776    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z" event={"ID":"f01e3592-abbe-44cb-bf71-5d3e039d4526","Type":"ContainerDied","Data":"38719961d5d18410f2a8590668104015c715df61f15febd018677d02b1d0ce70"}
Oct 31 19:19:39.069820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.069791    4484 scope.go:122] "RemoveContainer" containerID="4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20"
Oct 31 19:19:39.087190 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.087173    4484 scope.go:122] "RemoveContainer" containerID="4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20"
Oct 31 19:19:39.087500 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:19:39.087453    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20\": container with ID starting with 4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20 not found: ID does not exist" containerID="4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20"
Oct 31 19:19:39.087562 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.087509    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20"} err="failed to get container status \"4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20\": rpc error: code = NotFound desc = could not find container \"4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20\": container with ID starting with 4bbeb685bf61b281e92294d6c6079db73f1d0d18780bda27968127f7f1e82a20 not found: ID does not exist"
Oct 31 19:19:39.144596 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.144328    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:19:39.144596 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.144359    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htnj2\" (UniqueName: \"kubernetes.io/projected/f01e3592-abbe-44cb-bf71-5d3e039d4526-kube-api-access-htnj2\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:19:39.144596 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.144379    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01e3592-abbe-44cb-bf71-5d3e039d4526-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:19:39.144596 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.144394    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01e3592-abbe-44cb-bf71-5d3e039d4526-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:19:39.239234 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.239201    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"]
Oct 31 19:19:39.245255 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.245228    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d1ff32/auth-test-model-d1ff32-kserve-6dfc5f5557-bv25z"]
Oct 31 19:19:39.994504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:39.994445    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01e3592-abbe-44cb-bf71-5d3e039d4526" path="/var/lib/kubelet/pods/f01e3592-abbe-44cb-bf71-5d3e039d4526/volumes"
Oct 31 19:19:40.308547 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:40.308437    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:40.889924 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:40.889895    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:42.348444 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:42.348407    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:42.575599 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:42.575558    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:43.764696 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:43.764654    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:46.376559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:46.376526    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:46.588730 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:46.588687    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:49.072913 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:49.072881    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"
Oct 31 19:19:49.073330 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:49.072962    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"
Oct 31 19:19:49.110783 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:49.110745    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:49.670070 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:49.670018    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:49.843107 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:49.843069    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:50.058988 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.058959    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"]
Oct 31 19:19:50.059610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.059595    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f01e3592-abbe-44cb-bf71-5d3e039d4526" containerName="main"
Oct 31 19:19:50.059656 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.059611    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01e3592-abbe-44cb-bf71-5d3e039d4526" containerName="main"
Oct 31 19:19:50.059744 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.059736    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01e3592-abbe-44cb-bf71-5d3e039d4526" containerName="main"
Oct 31 19:19:50.064552 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.064527    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"
Oct 31 19:19:50.070600 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.070567    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b84878-41f2-4dfe-b207-431eff92d0ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4\" (UID: \"66b84878-41f2-4dfe-b207-431eff92d0ec\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"
Oct 31 19:19:50.071157 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.071116    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdg48\" (UniqueName: \"kubernetes.io/projected/66b84878-41f2-4dfe-b207-431eff92d0ec-kube-api-access-qdg48\") pod \"payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4\" (UID: \"66b84878-41f2-4dfe-b207-431eff92d0ec\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"
Oct 31 19:19:50.123949 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.123912    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"]
Oct 31 19:19:50.128042 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.128012    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-757472-dockercfg-xbtl7\""
Oct 31 19:19:50.128258 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.128013    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-757472\""
Oct 31 19:19:50.172881 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.172827    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b84878-41f2-4dfe-b207-431eff92d0ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4\" (UID: \"66b84878-41f2-4dfe-b207-431eff92d0ec\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"
Oct 31 19:19:50.174463 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.174336    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b84878-41f2-4dfe-b207-431eff92d0ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4\" (UID: \"66b84878-41f2-4dfe-b207-431eff92d0ec\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"
Oct 31 19:19:50.174754 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.174735    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdg48\" (UniqueName: \"kubernetes.io/projected/66b84878-41f2-4dfe-b207-431eff92d0ec-kube-api-access-qdg48\") pod \"payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4\" (UID: \"66b84878-41f2-4dfe-b207-431eff92d0ec\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"
Oct 31 19:19:50.201771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.201740    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"]
Oct 31 19:19:50.205972 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.205951    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"
Oct 31 19:19:50.235016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.234982    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdg48\" (UniqueName: \"kubernetes.io/projected/66b84878-41f2-4dfe-b207-431eff92d0ec-kube-api-access-qdg48\") pod \"payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4\" (UID: \"66b84878-41f2-4dfe-b207-431eff92d0ec\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"
Oct 31 19:19:50.269967 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.269937    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"]
Oct 31 19:19:50.276664 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.276629    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrrhh\" (UniqueName: \"kubernetes.io/projected/84a65681-76c3-4c44-83bc-65e439a6ecc8-kube-api-access-rrrhh\") pod \"payload-processing-e2e-autores-757472-85cd6bf685-kql5n\" (UID: \"84a65681-76c3-4c44-83bc-65e439a6ecc8\") " pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"
Oct 31 19:19:50.276850 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.276813    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a65681-76c3-4c44-83bc-65e439a6ecc8-plugins-config-volume\") pod \"payload-processing-e2e-autores-757472-85cd6bf685-kql5n\" (UID: \"84a65681-76c3-4c44-83bc-65e439a6ecc8\") " pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"
Oct 31 19:19:50.378658 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.378551    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a65681-76c3-4c44-83bc-65e439a6ecc8-plugins-config-volume\") pod \"payload-processing-e2e-autores-757472-85cd6bf685-kql5n\" (UID: \"84a65681-76c3-4c44-83bc-65e439a6ecc8\") " pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"
Oct 31 19:19:50.378882 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.378752    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrrhh\" (UniqueName: \"kubernetes.io/projected/84a65681-76c3-4c44-83bc-65e439a6ecc8-kube-api-access-rrrhh\") pod \"payload-processing-e2e-autores-757472-85cd6bf685-kql5n\" (UID: \"84a65681-76c3-4c44-83bc-65e439a6ecc8\") " pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"
Oct 31 19:19:50.378974 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.378871    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"
Oct 31 19:19:50.379373 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.379355    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a65681-76c3-4c44-83bc-65e439a6ecc8-plugins-config-volume\") pod \"payload-processing-e2e-autores-757472-85cd6bf685-kql5n\" (UID: \"84a65681-76c3-4c44-83bc-65e439a6ecc8\") " pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"
Oct 31 19:19:50.442429 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.442397    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrrhh\" (UniqueName: \"kubernetes.io/projected/84a65681-76c3-4c44-83bc-65e439a6ecc8-kube-api-access-rrrhh\") pod \"payload-processing-e2e-autores-757472-85cd6bf685-kql5n\" (UID: \"84a65681-76c3-4c44-83bc-65e439a6ecc8\") " pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"
Oct 31 19:19:50.519754 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.519721    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"
Oct 31 19:19:50.765178 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.765144    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"]
Oct 31 19:19:50.765691 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:19:50.765660    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b84878_41f2_4dfe_b207_431eff92d0ec.slice/crio-a8c6b5a54ef76b926dc4b35c7b5375e28b1b84dfcf33dcf10d32707df3cda7e7 WatchSource:0}: Error finding container a8c6b5a54ef76b926dc4b35c7b5375e28b1b84dfcf33dcf10d32707df3cda7e7: Status 404 returned error can't find the container with id a8c6b5a54ef76b926dc4b35c7b5375e28b1b84dfcf33dcf10d32707df3cda7e7
Oct 31 19:19:50.901897 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:50.901855    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"]
Oct 31 19:19:50.902751 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:19:50.902722    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a65681_76c3_4c44_83bc_65e439a6ecc8.slice/crio-cdb0cbd83854c981409773ec27c3dfbc86655a58461c6e03e6b3f37135b72b0e WatchSource:0}: Error finding container cdb0cbd83854c981409773ec27c3dfbc86655a58461c6e03e6b3f37135b72b0e: Status 404 returned error can't find the container with id cdb0cbd83854c981409773ec27c3dfbc86655a58461c6e03e6b3f37135b72b0e
Oct 31 19:19:51.203782 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:51.203681    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4" event={"ID":"66b84878-41f2-4dfe-b207-431eff92d0ec","Type":"ContainerStarted","Data":"33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705"}
Oct 31 19:19:51.204215 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:51.203783    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4" event={"ID":"66b84878-41f2-4dfe-b207-431eff92d0ec","Type":"ContainerStarted","Data":"a8c6b5a54ef76b926dc4b35c7b5375e28b1b84dfcf33dcf10d32707df3cda7e7"}
Oct 31 19:19:51.204215 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:51.203815    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"
Oct 31 19:19:51.205525 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:51.205503    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n" event={"ID":"84a65681-76c3-4c44-83bc-65e439a6ecc8","Type":"ContainerStarted","Data":"2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41"}
Oct 31 19:19:51.205610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:51.205542    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n" event={"ID":"84a65681-76c3-4c44-83bc-65e439a6ecc8","Type":"ContainerStarted","Data":"cdb0cbd83854c981409773ec27c3dfbc86655a58461c6e03e6b3f37135b72b0e"}
Oct 31 19:19:51.205610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:51.205578    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"
Oct 31 19:19:51.323835 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:51.323781    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4" podStartSLOduration=1.323766998 podStartE2EDuration="1.323766998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:51.32126875 +0000 UTC m=+3537.467718614" watchObservedRunningTime="2026-10-31 19:19:51.323766998 +0000 UTC m=+3537.470216862"
Oct 31 19:19:54.173045 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:54.173001    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:55.469695 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:19:55.469662    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:19:58.883284 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:19:58.883240    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:00.215849 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:00.215808    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:02.216152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:02.216125    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"
Oct 31 19:20:02.216579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:02.216190    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"
Oct 31 19:20:02.336176 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:02.336123    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n" podStartSLOduration=12.336107694 podStartE2EDuration="12.336107694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:51.43923509 +0000 UTC m=+3537.585684954" watchObservedRunningTime="2026-10-31 19:20:02.336107694 +0000 UTC m=+3548.482557556"
Oct 31 19:20:03.618279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:03.618243    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:04.659137 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:04.659093    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:08.218533 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:08.218439    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:08.435390 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:08.435356    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:08.601688 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:08.601648    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:12.797038 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:12.797003    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:20:13.185636 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:13.185534    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:17.976152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:17.976100    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:22.677410 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:22.677373    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:25.468271 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:20:25.468238    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:20:27.340120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:27.340059    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:27.500856 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:27.500811    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:32.247054 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:32.247014    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:32.471666 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:32.471621    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:32.640686 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:32.640597    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:37.180825 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:37.180794    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:41.461603 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.461568    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"]
Oct 31 19:20:41.466329 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.466305    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.520986 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.520944    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9cf479-openshift-default-dockercfg-9j62d\""
Oct 31 19:20:41.567460 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.567426    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-credential-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.567663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.567495    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13aca4a2-870e-4b0a-8162-043d154b8702-istiod-ca-cert\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.567663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.567586    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2t54\" (UniqueName: \"kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-kube-api-access-n2t54\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.567757 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.567719    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-data\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.567799 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.567785    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-envoy\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.567869 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.567854    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13aca4a2-870e-4b0a-8162-043d154b8702-istio-podinfo\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.567943 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.567921    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.568039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.568016    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-certs\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.568082 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.568057    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-istio-token\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.575467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.575443    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"]
Oct 31 19:20:41.669556 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.669521    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-data\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.669762 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.669573    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-envoy\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.669762 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.669636    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13aca4a2-870e-4b0a-8162-043d154b8702-istio-podinfo\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.669762 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.669652    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.669762 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.669681    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-certs\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.669762 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.669700    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-istio-token\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.669762 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.669761    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-credential-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.670097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.669796    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13aca4a2-870e-4b0a-8162-043d154b8702-istiod-ca-cert\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.670097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.669819    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2t54\" (UniqueName: \"kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-kube-api-access-n2t54\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.670097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.669981    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-data\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.670097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.670082    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-certs\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.670353 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.670246    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-credential-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.670451 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.670421    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-socket\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.670663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.670644    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13aca4a2-870e-4b0a-8162-043d154b8702-istiod-ca-cert\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.671898 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.671871    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-envoy\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.672130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.672114    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13aca4a2-870e-4b0a-8162-043d154b8702-istio-podinfo\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.728135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.728071    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-istio-token\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.728272 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.728207    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2t54\" (UniqueName: \"kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-kube-api-access-n2t54\") pod \"e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") " pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.781368 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.781340    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:41.973971 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:41.973934    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:42.213006 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:42.212971    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"]
Oct 31 19:20:42.213779 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:20:42.213750    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13aca4a2_870e_4b0a_8162_043d154b8702.slice/crio-b18117ee04bb26d29e091c4b7008dccd0a80a35f3b5067e89ba7791db3957489 WatchSource:0}: Error finding container b18117ee04bb26d29e091c4b7008dccd0a80a35f3b5067e89ba7791db3957489: Status 404 returned error can't find the container with id b18117ee04bb26d29e091c4b7008dccd0a80a35f3b5067e89ba7791db3957489
Oct 31 19:20:42.227147 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:42.227100    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:20:42.227272 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:42.227195    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:20:42.227272 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:42.227233    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:20:42.549833 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:42.549773    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp" event={"ID":"13aca4a2-870e-4b0a-8162-043d154b8702","Type":"ContainerStarted","Data":"f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384"}
Oct 31 19:20:42.550369 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:42.549889    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp" event={"ID":"13aca4a2-870e-4b0a-8162-043d154b8702","Type":"ContainerStarted","Data":"b18117ee04bb26d29e091c4b7008dccd0a80a35f3b5067e89ba7791db3957489"}
Oct 31 19:20:42.674350 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:42.674295    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp" podStartSLOduration=1.6742819249999998 podStartE2EDuration="1.674281925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:42.670690986 +0000 UTC m=+3588.817140864" watchObservedRunningTime="2026-10-31 19:20:42.674281925 +0000 UTC m=+3588.820731787"
Oct 31 19:20:42.786458 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:42.786423    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:43.785863 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:43.785823    4484 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:20:43.786300 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:43.785885    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp" podUID="13aca4a2-870e-4b0a-8162-043d154b8702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 31 19:20:44.785640 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:44.785601    4484 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:20:44.785834 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:44.785686    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp" podUID="13aca4a2-870e-4b0a-8162-043d154b8702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 31 19:20:45.784907 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:45.784865    4484 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:20:45.785580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:45.785545    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp" podUID="13aca4a2-870e-4b0a-8162-043d154b8702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 31 19:20:46.742103 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:46.742068    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:46.791497 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:46.787586    4484 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:20:46.791497 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:46.787645    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp" podUID="13aca4a2-870e-4b0a-8162-043d154b8702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 31 19:20:47.785658 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:47.785609    4484 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 31 19:20:47.785843 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:47.785687    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp" podUID="13aca4a2-870e-4b0a-8162-043d154b8702" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 31 19:20:48.786578 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:48.786546    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:48.787016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:48.786999    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:48.787746 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:48.787725    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:20:51.474489 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:51.474443    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:53.714667 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:53.714632    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:20:54.737183 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:54.737157    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:20:54.739557 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:54.739528    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:20:55.470367 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:20:55.470331    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:20:56.283104 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:20:56.283073    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:00.523138 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:00.523102    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:01.076877 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:01.076844    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:04.984291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:04.984262    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:05.489874 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.489843    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"]
Oct 31 19:21:05.494042 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.494026    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:05.549936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.549908    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"]
Oct 31 19:21:05.560155 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.560121    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-metrics-certs\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:05.560324 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.560192    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:05.560524 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.560466    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2d55ac-f4ea-48db-8e9a-42debccb1803-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:05.560700 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.560676    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7j2m\" (UniqueName: \"kubernetes.io/projected/7b2d55ac-f4ea-48db-8e9a-42debccb1803-kube-api-access-g7j2m\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:05.597200 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.597164    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-3744b0-serving-cert\""
Oct 31 19:21:05.661752 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.661716    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2d55ac-f4ea-48db-8e9a-42debccb1803-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:05.661952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.661764    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7j2m\" (UniqueName: \"kubernetes.io/projected/7b2d55ac-f4ea-48db-8e9a-42debccb1803-kube-api-access-g7j2m\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:05.661952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.661828    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-metrics-certs\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:05.662333 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.661876    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:05.662617 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:21:05.662005    4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-serving-cert: secret "maas-api-e2e-extmod-b-3744b0-serving-cert" not found
Oct 31 19:21:05.662791 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:21:05.662671    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-maas-api-tls podName:7b2d55ac-f4ea-48db-8e9a-42debccb1803 nodeName:}" failed. No retries permitted until 2026-10-31 19:21:06.162649521 +0000 UTC m=+3612.309099365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-maas-api-tls") pod "maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb" (UID: "7b2d55ac-f4ea-48db-8e9a-42debccb1803") : secret "maas-api-e2e-extmod-b-3744b0-serving-cert" not found
Oct 31 19:21:05.662791 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.662486    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2d55ac-f4ea-48db-8e9a-42debccb1803-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:05.664680 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.664661    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-metrics-certs\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:05.707628 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.707589    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:05.720703 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.720673    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7j2m\" (UniqueName: \"kubernetes.io/projected/7b2d55ac-f4ea-48db-8e9a-42debccb1803-kube-api-access-g7j2m\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:05.872709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:05.872624    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:06.023211 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:06.023174    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:06.168246 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:06.168153    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:06.170900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:06.170873    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:06.407939 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:06.407909    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:06.790303 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:06.790277    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"]
Oct 31 19:21:06.791289 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:21:06.791260    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2d55ac_f4ea_48db_8e9a_42debccb1803.slice/crio-423ab42f4ae893bb6141257715c14f424b347c1ce600e7fd468d82585bee2437 WatchSource:0}: Error finding container 423ab42f4ae893bb6141257715c14f424b347c1ce600e7fd468d82585bee2437: Status 404 returned error can't find the container with id 423ab42f4ae893bb6141257715c14f424b347c1ce600e7fd468d82585bee2437
Oct 31 19:21:06.793012 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:06.792993    4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:21:07.736121 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:07.736087    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb" event={"ID":"7b2d55ac-f4ea-48db-8e9a-42debccb1803","Type":"ContainerStarted","Data":"a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4"}
Oct 31 19:21:07.736504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:07.736132    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb" event={"ID":"7b2d55ac-f4ea-48db-8e9a-42debccb1803","Type":"ContainerStarted","Data":"423ab42f4ae893bb6141257715c14f424b347c1ce600e7fd468d82585bee2437"}
Oct 31 19:21:07.736504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:07.736223    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:07.853381 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:07.853323    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb" podStartSLOduration=2.011483789 podStartE2EDuration="2.853302357s" totalImagesPullingTime="841.818568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:21:07.851848423 +0000 UTC m=+3613.998298286" watchObservedRunningTime="2026-10-31 19:21:07.853302357 +0000 UTC m=+3613.999752220"
Oct 31 19:21:10.550764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:10.550728    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:13.749642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:13.749610    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:21:15.241276 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:15.241230    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:19.970627 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:19.970596    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:24.595005 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:24.594964    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:24.754774 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:24.754743    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:24.922806 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:24.922699    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:25.469621 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:21:25.469582    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:21:29.491029 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:29.490992    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:34.188089 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:34.188053    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:38.772308 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:38.772223    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:39.595919 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:39.595890    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:21:43.487468 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:43.487434    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:44.003073 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:44.003037    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:48.100249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:48.100210    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:49.484877 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:49.484833    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:49.661111 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:49.661070    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:50.337043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:50.337013    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:51.530632 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:51.530599    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:52.748879 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:52.748850    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:55.468752 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:21:55.468712    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:21:55.806700 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:55.806662    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:57.463069 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:57.463030    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:58.980388 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:21:58.980334    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:00.853707 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:00.853664    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:02.100960 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:02.100924    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:03.135098 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:03.135045    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:04.454257 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:04.454226    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:04.661037 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:04.660993    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:05.263076 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:05.263040    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:05.423926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:05.423892    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:06.973325 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:06.973279    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:08.973097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:08.973057    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:09.977680 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:09.977644    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:10.140133 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:10.140071    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:11.098235 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:11.098202    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:11.255942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:11.255907    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:12.953073 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:12.953023    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:14.708995 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:14.708944    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:18.033854 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:18.033823    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:22.816572 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:22.816532    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:25.468983 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:22:25.468947    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:22:27.458232 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:27.458188    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:32.141449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:32.141408    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:36.787230 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:36.787191    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:36.990262 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:36.990223    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:39.024291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:39.024256    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:22:41.492789 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:41.492746    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:43.515361 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:43.515318    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:43.729613 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:43.729580    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"]
Oct 31 19:22:43.849886 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:43.849788    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"]
Oct 31 19:22:44.193369 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:44.193276    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:44.411926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:44.411891    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:45.721396 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:45.721356    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:46.621966 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:46.621925    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:48.730703 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:48.730665    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r" podUID="13a2c839-c270-412a-a636-17ee576a7c7f" containerName="payload-processing" containerID="cri-o://809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb" gracePeriod=25
Oct 31 19:22:48.850855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:48.850800    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99" podUID="85defdb9-17ce-48a9-8785-0171ab21881a" containerName="payload-pre-processing" containerID="cri-o://8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a" gracePeriod=25
Oct 31 19:22:49.013518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.013467    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"
Oct 31 19:22:49.138950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.138923    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"
Oct 31 19:22:49.139673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.139647    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13a2c839-c270-412a-a636-17ee576a7c7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a2c839-c270-412a-a636-17ee576a7c7f-plugins-config-volume\") pod \"13a2c839-c270-412a-a636-17ee576a7c7f\" (UID: \"13a2c839-c270-412a-a636-17ee576a7c7f\") "
Oct 31 19:22:49.139811 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.139694    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13a2c839-c270-412a-a636-17ee576a7c7f-kube-api-access-wtpkb\" (UniqueName: \"kubernetes.io/projected/13a2c839-c270-412a-a636-17ee576a7c7f-kube-api-access-wtpkb\") pod \"13a2c839-c270-412a-a636-17ee576a7c7f\" (UID: \"13a2c839-c270-412a-a636-17ee576a7c7f\") "
Oct 31 19:22:49.140252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.140066    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a2c839-c270-412a-a636-17ee576a7c7f-plugins-config-volume" pod "13a2c839-c270-412a-a636-17ee576a7c7f" (UID: "13a2c839-c270-412a-a636-17ee576a7c7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:22:49.140343 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.140303    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13a2c839-c270-412a-a636-17ee576a7c7f-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:22:49.142715 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.142684    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a2c839-c270-412a-a636-17ee576a7c7f-kube-api-access-wtpkb" pod "13a2c839-c270-412a-a636-17ee576a7c7f" (UID: "13a2c839-c270-412a-a636-17ee576a7c7f"). InnerVolumeSpecName "kube-api-access-wtpkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:22:49.241813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.241724    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85defdb9-17ce-48a9-8785-0171ab21881a-kube-api-access-fpzb2\" (UniqueName: \"kubernetes.io/projected/85defdb9-17ce-48a9-8785-0171ab21881a-kube-api-access-fpzb2\") pod \"85defdb9-17ce-48a9-8785-0171ab21881a\" (UID: \"85defdb9-17ce-48a9-8785-0171ab21881a\") "
Oct 31 19:22:49.241998 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.241857    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85defdb9-17ce-48a9-8785-0171ab21881a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85defdb9-17ce-48a9-8785-0171ab21881a-plugins-config-volume\") pod \"85defdb9-17ce-48a9-8785-0171ab21881a\" (UID: \"85defdb9-17ce-48a9-8785-0171ab21881a\") "
Oct 31 19:22:49.242270 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.242244    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85defdb9-17ce-48a9-8785-0171ab21881a-plugins-config-volume" pod "85defdb9-17ce-48a9-8785-0171ab21881a" (UID: "85defdb9-17ce-48a9-8785-0171ab21881a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:22:49.242386 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.242318    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtpkb\" (UniqueName: \"kubernetes.io/projected/13a2c839-c270-412a-a636-17ee576a7c7f-kube-api-access-wtpkb\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:22:49.244065 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.244033    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85defdb9-17ce-48a9-8785-0171ab21881a-kube-api-access-fpzb2" pod "85defdb9-17ce-48a9-8785-0171ab21881a" (UID: "85defdb9-17ce-48a9-8785-0171ab21881a"). InnerVolumeSpecName "kube-api-access-fpzb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:22:49.258468 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.258435    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:49.343669 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.343633    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85defdb9-17ce-48a9-8785-0171ab21881a-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:22:49.343669 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.343663    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpzb2\" (UniqueName: \"kubernetes.io/projected/85defdb9-17ce-48a9-8785-0171ab21881a-kube-api-access-fpzb2\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:22:49.403637 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.403600    4484 generic.go:356] "Generic (PLEG): container finished" podID="13a2c839-c270-412a-a636-17ee576a7c7f" containerID="809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb" exitCode=0
Oct 31 19:22:49.403832 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.403647    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"
Oct 31 19:22:49.403832 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.403688    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r" event={"ID":"13a2c839-c270-412a-a636-17ee576a7c7f","Type":"ContainerDied","Data":"809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb"}
Oct 31 19:22:49.403832 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.403740    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r" event={"ID":"13a2c839-c270-412a-a636-17ee576a7c7f","Type":"ContainerDied","Data":"7deffbb901ed4af87db8c7250063789c9da8f31637327fbaa73f877f0fc282fc"}
Oct 31 19:22:49.403832 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.403769    4484 scope.go:122] "RemoveContainer" containerID="809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb"
Oct 31 19:22:49.405583 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.405537    4484 generic.go:356] "Generic (PLEG): container finished" podID="85defdb9-17ce-48a9-8785-0171ab21881a" containerID="8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a" exitCode=0
Oct 31 19:22:49.405717 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.405590    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"
Oct 31 19:22:49.405717 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.405625    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99" event={"ID":"85defdb9-17ce-48a9-8785-0171ab21881a","Type":"ContainerDied","Data":"8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a"}
Oct 31 19:22:49.405717 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.405684    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99" event={"ID":"85defdb9-17ce-48a9-8785-0171ab21881a","Type":"ContainerDied","Data":"c6eda40e03ee5473bda08b1159d2147dd2088b1c2cff6e0c2144a99cd26be695"}
Oct 31 19:22:49.414413 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.414321    4484 scope.go:122] "RemoveContainer" containerID="809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb"
Oct 31 19:22:49.414706 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:22:49.414685    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb\": container with ID starting with 809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb not found: ID does not exist" containerID="809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb"
Oct 31 19:22:49.414797 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.414714    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb"} err="failed to get container status \"809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb\": rpc error: code = NotFound desc = could not find container \"809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb\": container with ID starting with 809839cd706a11bcf2904c30d75dd372754af60e3679b970ed0fc8c507a509eb not found: ID does not exist"
Oct 31 19:22:49.414797 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.414734    4484 scope.go:122] "RemoveContainer" containerID="8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a"
Oct 31 19:22:49.426596 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.426576    4484 scope.go:122] "RemoveContainer" containerID="8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a"
Oct 31 19:22:49.426892 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:22:49.426871    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a\": container with ID starting with 8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a not found: ID does not exist" containerID="8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a"
Oct 31 19:22:49.426956 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.426900    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a"} err="failed to get container status \"8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a\": rpc error: code = NotFound desc = could not find container \"8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a\": container with ID starting with 8bdddb60526c5e3ad82d0d734713a78fcf02f4703b78418c496cb7c4cc77b22a not found: ID does not exist"
Oct 31 19:22:49.572056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.571968    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"]
Oct 31 19:22:49.574810 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.574787    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-75c484-5b79c988f5-qjt99"]
Oct 31 19:22:49.691583 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.691550    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"]
Oct 31 19:22:49.697110 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.697081    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-75c484-7746966cbf-l484r"]
Oct 31 19:22:49.994610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.994582    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13a2c839-c270-412a-a636-17ee576a7c7f" path="/var/lib/kubelet/pods/13a2c839-c270-412a-a636-17ee576a7c7f/volumes"
Oct 31 19:22:49.995022 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:49.995007    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85defdb9-17ce-48a9-8785-0171ab21881a" path="/var/lib/kubelet/pods/85defdb9-17ce-48a9-8785-0171ab21881a/volumes"
Oct 31 19:22:50.795310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:50.795277    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"]
Oct 31 19:22:50.905340 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:50.905298    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"]
Oct 31 19:22:51.185384 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:51.185286    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:51.465636 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:51.465508    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:51.636042 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:51.636006    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:52.047501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:52.047440    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:55.469142 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:22:55.469103    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:22:55.796063 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:55.796012    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n" podUID="84a65681-76c3-4c44-83bc-65e439a6ecc8" containerName="payload-processing" containerID="cri-o://2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41" gracePeriod=25
Oct 31 19:22:55.905648 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:55.905598    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4" podUID="66b84878-41f2-4dfe-b207-431eff92d0ec" containerName="payload-pre-processing" containerID="cri-o://33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705" gracePeriod=25
Oct 31 19:22:56.078604 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.078575    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"
Oct 31 19:22:56.160717 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.160680    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:56.199735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.199710    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"
Oct 31 19:22:56.228241 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.228193    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a65681-76c3-4c44-83bc-65e439a6ecc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a65681-76c3-4c44-83bc-65e439a6ecc8-plugins-config-volume\") pod \"84a65681-76c3-4c44-83bc-65e439a6ecc8\" (UID: \"84a65681-76c3-4c44-83bc-65e439a6ecc8\") "
Oct 31 19:22:56.228241 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.228243    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a65681-76c3-4c44-83bc-65e439a6ecc8-kube-api-access-rrrhh\" (UniqueName: \"kubernetes.io/projected/84a65681-76c3-4c44-83bc-65e439a6ecc8-kube-api-access-rrrhh\") pod \"84a65681-76c3-4c44-83bc-65e439a6ecc8\" (UID: \"84a65681-76c3-4c44-83bc-65e439a6ecc8\") "
Oct 31 19:22:56.228667 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.228640    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a65681-76c3-4c44-83bc-65e439a6ecc8-plugins-config-volume" pod "84a65681-76c3-4c44-83bc-65e439a6ecc8" (UID: "84a65681-76c3-4c44-83bc-65e439a6ecc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:22:56.228861 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.228835    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84a65681-76c3-4c44-83bc-65e439a6ecc8-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:22:56.230859 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.230838    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a65681-76c3-4c44-83bc-65e439a6ecc8-kube-api-access-rrrhh" pod "84a65681-76c3-4c44-83bc-65e439a6ecc8" (UID: "84a65681-76c3-4c44-83bc-65e439a6ecc8"). InnerVolumeSpecName "kube-api-access-rrrhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:22:56.329622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.329542    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b84878-41f2-4dfe-b207-431eff92d0ec-kube-api-access-qdg48\" (UniqueName: \"kubernetes.io/projected/66b84878-41f2-4dfe-b207-431eff92d0ec-kube-api-access-qdg48\") pod \"66b84878-41f2-4dfe-b207-431eff92d0ec\" (UID: \"66b84878-41f2-4dfe-b207-431eff92d0ec\") "
Oct 31 19:22:56.329622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.329585    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b84878-41f2-4dfe-b207-431eff92d0ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b84878-41f2-4dfe-b207-431eff92d0ec-plugins-config-volume\") pod \"66b84878-41f2-4dfe-b207-431eff92d0ec\" (UID: \"66b84878-41f2-4dfe-b207-431eff92d0ec\") "
Oct 31 19:22:56.330026 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.330001    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b84878-41f2-4dfe-b207-431eff92d0ec-plugins-config-volume" pod "66b84878-41f2-4dfe-b207-431eff92d0ec" (UID: "66b84878-41f2-4dfe-b207-431eff92d0ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:22:56.330221 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.330203    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b84878-41f2-4dfe-b207-431eff92d0ec-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:22:56.330265 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.330226    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrrhh\" (UniqueName: \"kubernetes.io/projected/84a65681-76c3-4c44-83bc-65e439a6ecc8-kube-api-access-rrrhh\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:22:56.331900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.331877    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b84878-41f2-4dfe-b207-431eff92d0ec-kube-api-access-qdg48" pod "66b84878-41f2-4dfe-b207-431eff92d0ec" (UID: "66b84878-41f2-4dfe-b207-431eff92d0ec"). InnerVolumeSpecName "kube-api-access-qdg48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:22:56.431922 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.431885    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdg48\" (UniqueName: \"kubernetes.io/projected/66b84878-41f2-4dfe-b207-431eff92d0ec-kube-api-access-qdg48\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:22:56.455911 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.455878    4484 generic.go:356] "Generic (PLEG): container finished" podID="84a65681-76c3-4c44-83bc-65e439a6ecc8" containerID="2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41" exitCode=0
Oct 31 19:22:56.456056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.455939    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"
Oct 31 19:22:56.456056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.455962    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n" event={"ID":"84a65681-76c3-4c44-83bc-65e439a6ecc8","Type":"ContainerDied","Data":"2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41"}
Oct 31 19:22:56.456056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.456011    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n" event={"ID":"84a65681-76c3-4c44-83bc-65e439a6ecc8","Type":"ContainerDied","Data":"cdb0cbd83854c981409773ec27c3dfbc86655a58461c6e03e6b3f37135b72b0e"}
Oct 31 19:22:56.456056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.456033    4484 scope.go:122] "RemoveContainer" containerID="2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41"
Oct 31 19:22:56.457586 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.457565    4484 generic.go:356] "Generic (PLEG): container finished" podID="66b84878-41f2-4dfe-b207-431eff92d0ec" containerID="33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705" exitCode=0
Oct 31 19:22:56.457693 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.457660    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"
Oct 31 19:22:56.457693 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.457669    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4" event={"ID":"66b84878-41f2-4dfe-b207-431eff92d0ec","Type":"ContainerDied","Data":"33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705"}
Oct 31 19:22:56.457822 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.457700    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4" event={"ID":"66b84878-41f2-4dfe-b207-431eff92d0ec","Type":"ContainerDied","Data":"a8c6b5a54ef76b926dc4b35c7b5375e28b1b84dfcf33dcf10d32707df3cda7e7"}
Oct 31 19:22:56.470073 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.470058    4484 scope.go:122] "RemoveContainer" containerID="2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41"
Oct 31 19:22:56.470369 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:22:56.470333    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41\": container with ID starting with 2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41 not found: ID does not exist" containerID="2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41"
Oct 31 19:22:56.470369 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.470355    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41"} err="failed to get container status \"2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41\": rpc error: code = NotFound desc = could not find container \"2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41\": container with ID starting with 2ade0aa79ff9d3b4936cbf528f2a8d6dfe498c330e4fbf4fe9af7ad640d41a41 not found: ID does not exist"
Oct 31 19:22:56.470456 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.470372    4484 scope.go:122] "RemoveContainer" containerID="33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705"
Oct 31 19:22:56.479374 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.479354    4484 scope.go:122] "RemoveContainer" containerID="33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705"
Oct 31 19:22:56.479636 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:22:56.479617    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705\": container with ID starting with 33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705 not found: ID does not exist" containerID="33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705"
Oct 31 19:22:56.479708 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.479643    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705"} err="failed to get container status \"33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705\": rpc error: code = NotFound desc = could not find container \"33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705\": container with ID starting with 33b02f198af29fbee2bbba5c49f26ae4851394c0c19b15ba9feb25379243c705 not found: ID does not exist"
Oct 31 19:22:56.627039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.626958    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"]
Oct 31 19:22:56.632307 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.632279    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-757472-b6cbdf7b7-xbmd4"]
Oct 31 19:22:56.745278 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.745245    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"]
Oct 31 19:22:56.747984 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:56.747964    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-757472-85cd6bf685-kql5n"]
Oct 31 19:22:57.966347 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:57.966312    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:57.994281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:57.994251    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b84878-41f2-4dfe-b207-431eff92d0ec" path="/var/lib/kubelet/pods/66b84878-41f2-4dfe-b207-431eff92d0ec/volumes"
Oct 31 19:22:57.994756 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:22:57.994742    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a65681-76c3-4c44-83bc-65e439a6ecc8" path="/var/lib/kubelet/pods/84a65681-76c3-4c44-83bc-65e439a6ecc8/volumes"
Oct 31 19:23:00.436600 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:00.436566    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:01.210998 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:01.210967    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:02.204951 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:02.204915    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:02.588689 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:02.588653    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:04.377391 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:04.377351    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:04.565413 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:04.565378    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:05.662061 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:05.661980    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:05.948148 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:05.948061    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:06.585364 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:06.585323    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:07.347499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:07.347442    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:08.521542 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:08.521507    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:09.200904 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:09.200871    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:09.443107 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:09.443069    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:12.690839 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:12.690744    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:13.645138 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:13.645102    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:13.889823 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:13.889789    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:16.966516 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:16.966457    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:17.229689 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:17.229606    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:17.508598 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:17.508561    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:19.513960 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:19.513923    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:21.264027 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:21.263989    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:21.558485 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:21.558386    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:22.956358 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:22.956320    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:23.073797 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:23.073760    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:23:23.535374 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:23.535328    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:24.277194 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:24.277157    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:24.786795 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:24.786751    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:25.469507 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:23:25.469459    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:23:29.848440 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:29.848409    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:30.769872 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:30.769841    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:31.090014 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:31.089925    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:31.293844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:31.293805    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:34.294324 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:34.294293    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:37.169391 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:37.169354    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:37.348039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:37.348001    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:37.816999 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:37.816965    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:38.716607 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:38.716572    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:39.212115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:39.212079    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:41.149872 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:41.149834    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:41.345579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:41.345546    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:42.508522 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:42.508459    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:43.458375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:43.458344    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:43.654373 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:43.654339    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:43.833547 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:43.833509    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:44.721405 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:44.721365    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:44.943401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:44.943360    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:45.232951 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:45.232915    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:47.419442 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:47.419407    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:51.741713 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:51.741674    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:53.682813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:53.682766    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:55.471810 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:23:55.471767    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:23:58.791036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:23:58.790998    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:00.108725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:00.108687    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:00.417291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:00.417189    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:01.586272 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:01.586236    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:03.461635 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:03.461593    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:03.666516 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:03.666446    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:03.882052 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:03.882017    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:05.095327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:05.095281    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:05.560734 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:05.560693    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:06.151594 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:06.151560    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:06.493789 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:06.493728    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:09.002535 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:09.002503    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:12.358382 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:12.358339    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:13.385463 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:13.385430    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"]
Oct 31 19:24:13.494670 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:13.494642    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"]
Oct 31 19:24:13.775030 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:13.774986    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:14.008183 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:14.008137    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:14.280488 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:14.280453    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:18.386380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.386333    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k" podUID="108076ec-b5aa-4b14-bed6-70e3714f2a1e" containerName="payload-processing" containerID="cri-o://cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f" gracePeriod=25
Oct 31 19:24:18.454816 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.454784    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:18.495681 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.495627    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g" podUID="1496d6eb-f023-4dfe-8dfc-7b8667661896" containerName="payload-pre-processing" containerID="cri-o://49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d" gracePeriod=25
Oct 31 19:24:18.659926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.659903    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"
Oct 31 19:24:18.735836 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.735805    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108076ec-b5aa-4b14-bed6-70e3714f2a1e-kube-api-access-c86b6\" (UniqueName: \"kubernetes.io/projected/108076ec-b5aa-4b14-bed6-70e3714f2a1e-kube-api-access-c86b6\") pod \"108076ec-b5aa-4b14-bed6-70e3714f2a1e\" (UID: \"108076ec-b5aa-4b14-bed6-70e3714f2a1e\") "
Oct 31 19:24:18.736040 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.735852    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108076ec-b5aa-4b14-bed6-70e3714f2a1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108076ec-b5aa-4b14-bed6-70e3714f2a1e-plugins-config-volume\") pod \"108076ec-b5aa-4b14-bed6-70e3714f2a1e\" (UID: \"108076ec-b5aa-4b14-bed6-70e3714f2a1e\") "
Oct 31 19:24:18.736354 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.736326    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108076ec-b5aa-4b14-bed6-70e3714f2a1e-plugins-config-volume" pod "108076ec-b5aa-4b14-bed6-70e3714f2a1e" (UID: "108076ec-b5aa-4b14-bed6-70e3714f2a1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:24:18.738334 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.738305    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108076ec-b5aa-4b14-bed6-70e3714f2a1e-kube-api-access-c86b6" pod "108076ec-b5aa-4b14-bed6-70e3714f2a1e" (UID: "108076ec-b5aa-4b14-bed6-70e3714f2a1e"). InnerVolumeSpecName "kube-api-access-c86b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:18.759546 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.759523    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"
Oct 31 19:24:18.837255 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.837222    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1496d6eb-f023-4dfe-8dfc-7b8667661896-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1496d6eb-f023-4dfe-8dfc-7b8667661896-plugins-config-volume\") pod \"1496d6eb-f023-4dfe-8dfc-7b8667661896\" (UID: \"1496d6eb-f023-4dfe-8dfc-7b8667661896\") "
Oct 31 19:24:18.837420 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.837345    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1496d6eb-f023-4dfe-8dfc-7b8667661896-kube-api-access-p5rpj\" (UniqueName: \"kubernetes.io/projected/1496d6eb-f023-4dfe-8dfc-7b8667661896-kube-api-access-p5rpj\") pod \"1496d6eb-f023-4dfe-8dfc-7b8667661896\" (UID: \"1496d6eb-f023-4dfe-8dfc-7b8667661896\") "
Oct 31 19:24:18.837696 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.837672    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1496d6eb-f023-4dfe-8dfc-7b8667661896-plugins-config-volume" pod "1496d6eb-f023-4dfe-8dfc-7b8667661896" (UID: "1496d6eb-f023-4dfe-8dfc-7b8667661896"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:24:18.837879 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.837863    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1496d6eb-f023-4dfe-8dfc-7b8667661896-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:18.837924 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.837883    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c86b6\" (UniqueName: \"kubernetes.io/projected/108076ec-b5aa-4b14-bed6-70e3714f2a1e-kube-api-access-c86b6\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:18.837924 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.837894    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108076ec-b5aa-4b14-bed6-70e3714f2a1e-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:18.839547 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.839526    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1496d6eb-f023-4dfe-8dfc-7b8667661896-kube-api-access-p5rpj" pod "1496d6eb-f023-4dfe-8dfc-7b8667661896" (UID: "1496d6eb-f023-4dfe-8dfc-7b8667661896"). InnerVolumeSpecName "kube-api-access-p5rpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:18.938736 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:18.938668    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rpj\" (UniqueName: \"kubernetes.io/projected/1496d6eb-f023-4dfe-8dfc-7b8667661896-kube-api-access-p5rpj\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:19.013428 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.013397    4484 generic.go:356] "Generic (PLEG): container finished" podID="1496d6eb-f023-4dfe-8dfc-7b8667661896" containerID="49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d" exitCode=0
Oct 31 19:24:19.013657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.013444    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"
Oct 31 19:24:19.013657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.013496    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g" event={"ID":"1496d6eb-f023-4dfe-8dfc-7b8667661896","Type":"ContainerDied","Data":"49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d"}
Oct 31 19:24:19.013657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.013543    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g" event={"ID":"1496d6eb-f023-4dfe-8dfc-7b8667661896","Type":"ContainerDied","Data":"6398da713d69886d3165a2279cf4d4ca5567644341f7b9b874e71fac28c7e9b8"}
Oct 31 19:24:19.013657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.013567    4484 scope.go:122] "RemoveContainer" containerID="49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d"
Oct 31 19:24:19.014926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.014901    4484 generic.go:356] "Generic (PLEG): container finished" podID="108076ec-b5aa-4b14-bed6-70e3714f2a1e" containerID="cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f" exitCode=0
Oct 31 19:24:19.015072 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.014950    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"
Oct 31 19:24:19.015072 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.014985    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k" event={"ID":"108076ec-b5aa-4b14-bed6-70e3714f2a1e","Type":"ContainerDied","Data":"cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f"}
Oct 31 19:24:19.015072 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.015025    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k" event={"ID":"108076ec-b5aa-4b14-bed6-70e3714f2a1e","Type":"ContainerDied","Data":"392f86cf38e9b2f0d80a1b9d23a770237e1e381cc76c4463bde0223411835daf"}
Oct 31 19:24:19.025608 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.025590    4484 scope.go:122] "RemoveContainer" containerID="49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d"
Oct 31 19:24:19.025908 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:24:19.025889    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d\": container with ID starting with 49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d not found: ID does not exist" containerID="49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d"
Oct 31 19:24:19.025990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.025915    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d"} err="failed to get container status \"49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d\": rpc error: code = NotFound desc = could not find container \"49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d\": container with ID starting with 49bfe951b946193d31b577c88d64cb403bbb93994434cfa7313566617017a78d not found: ID does not exist"
Oct 31 19:24:19.025990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.025933    4484 scope.go:122] "RemoveContainer" containerID="cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f"
Oct 31 19:24:19.035167 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.035147    4484 scope.go:122] "RemoveContainer" containerID="cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f"
Oct 31 19:24:19.035416 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:24:19.035398    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f\": container with ID starting with cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f not found: ID does not exist" containerID="cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f"
Oct 31 19:24:19.035501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.035423    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f"} err="failed to get container status \"cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f\": rpc error: code = NotFound desc = could not find container \"cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f\": container with ID starting with cff4bb5ffef761f10822cf93ddc997d2e950e6b8e3f7c965f9d537665a00757f not found: ID does not exist"
Oct 31 19:24:19.157075 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.157046    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:19.188936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.187565    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"]
Oct 31 19:24:19.194302 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.194269    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3744b0-7d6c4d5cc7-nt45k"]
Oct 31 19:24:19.303892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.303859    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"]
Oct 31 19:24:19.306975 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.306949    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3744b0-777cd76455-f6h4g"]
Oct 31 19:24:19.994631 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.994601    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108076ec-b5aa-4b14-bed6-70e3714f2a1e" path="/var/lib/kubelet/pods/108076ec-b5aa-4b14-bed6-70e3714f2a1e/volumes"
Oct 31 19:24:19.995020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:19.994986    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1496d6eb-f023-4dfe-8dfc-7b8667661896" path="/var/lib/kubelet/pods/1496d6eb-f023-4dfe-8dfc-7b8667661896/volumes"
Oct 31 19:24:24.076465 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:24.076422    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:25.432591 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.432549    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"]
Oct 31 19:24:25.433354 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433336    4484 cpu_manager.go:414] "containerMap: removing container" podUID="108076ec-b5aa-4b14-bed6-70e3714f2a1e" containerName="payload-processing"
Oct 31 19:24:25.433354 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433355    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108076ec-b5aa-4b14-bed6-70e3714f2a1e" containerName="payload-processing"
Oct 31 19:24:25.433450 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433369    4484 cpu_manager.go:414] "containerMap: removing container" podUID="1496d6eb-f023-4dfe-8dfc-7b8667661896" containerName="payload-pre-processing"
Oct 31 19:24:25.433450 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433375    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1496d6eb-f023-4dfe-8dfc-7b8667661896" containerName="payload-pre-processing"
Oct 31 19:24:25.433450 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433408    4484 cpu_manager.go:414] "containerMap: removing container" podUID="13a2c839-c270-412a-a636-17ee576a7c7f" containerName="payload-processing"
Oct 31 19:24:25.433450 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433413    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13a2c839-c270-412a-a636-17ee576a7c7f" containerName="payload-processing"
Oct 31 19:24:25.433450 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433425    4484 cpu_manager.go:414] "containerMap: removing container" podUID="66b84878-41f2-4dfe-b207-431eff92d0ec" containerName="payload-pre-processing"
Oct 31 19:24:25.433450 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433430    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b84878-41f2-4dfe-b207-431eff92d0ec" containerName="payload-pre-processing"
Oct 31 19:24:25.433683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433459    4484 cpu_manager.go:414] "containerMap: removing container" podUID="85defdb9-17ce-48a9-8785-0171ab21881a" containerName="payload-pre-processing"
Oct 31 19:24:25.433683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433465    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85defdb9-17ce-48a9-8785-0171ab21881a" containerName="payload-pre-processing"
Oct 31 19:24:25.433683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433495    4484 cpu_manager.go:414] "containerMap: removing container" podUID="84a65681-76c3-4c44-83bc-65e439a6ecc8" containerName="payload-processing"
Oct 31 19:24:25.433683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433500    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a65681-76c3-4c44-83bc-65e439a6ecc8" containerName="payload-processing"
Oct 31 19:24:25.433683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433603    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="85defdb9-17ce-48a9-8785-0171ab21881a" containerName="payload-pre-processing"
Oct 31 19:24:25.433683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433609    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a65681-76c3-4c44-83bc-65e439a6ecc8" containerName="payload-processing"
Oct 31 19:24:25.433683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433619    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="108076ec-b5aa-4b14-bed6-70e3714f2a1e" containerName="payload-processing"
Oct 31 19:24:25.433683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433624    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1496d6eb-f023-4dfe-8dfc-7b8667661896" containerName="payload-pre-processing"
Oct 31 19:24:25.433683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433635    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="13a2c839-c270-412a-a636-17ee576a7c7f" containerName="payload-processing"
Oct 31 19:24:25.433683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.433640    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b84878-41f2-4dfe-b207-431eff92d0ec" containerName="payload-pre-processing"
Oct 31 19:24:25.438349 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.438331    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.470150 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:24:25.470114    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:24:25.547297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.547262    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"]
Oct 31 19:24:25.587859 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.587820    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nsw28\""
Oct 31 19:24:25.614960 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.614926    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.615146 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.614979    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.615146 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.615070    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.615146 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.615103    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.615345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.615170    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.615345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.615302    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk9t4\" (UniqueName: \"kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-kube-api-access-dk9t4\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.615439 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.615361    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.615439 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.615401    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbcf347-7edf-499d-87ea-4ea645f2631e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.615543 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.615510    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.716375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.716274    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9t4\" (UniqueName: \"kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-kube-api-access-dk9t4\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.716375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.716326    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.716375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.716369    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbcf347-7edf-499d-87ea-4ea645f2631e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.716708 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.716389    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.716708 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.716449    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.716708 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.716499    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.716708 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.716547    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.716708 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.716583    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.716708 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.716655    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.717020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.716856    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.717020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.716992    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.717091 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.717034    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.717577 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.717554    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.717685 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.717619    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbcf347-7edf-499d-87ea-4ea645f2631e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.719185 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.719164    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.719905 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.719884    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.775590 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.775551    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk9t4\" (UniqueName: \"kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-kube-api-access-dk9t4\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:25.775742 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:25.775642    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2gq6t\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:26.053198 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:26.053166    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:26.485350 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:26.485268    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"]
Oct 31 19:24:26.521827 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:24:26.521799    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbcf347_7edf_499d_87ea_4ea645f2631e.slice/crio-4abd5e356820ab6069039c68988be129aafc67e76e5fe5ed373aca51ba5c17a1 WatchSource:0}: Error finding container 4abd5e356820ab6069039c68988be129aafc67e76e5fe5ed373aca51ba5c17a1: Status 404 returned error can't find the container with id 4abd5e356820ab6069039c68988be129aafc67e76e5fe5ed373aca51ba5c17a1
Oct 31 19:24:26.524346 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:26.524311    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:24:26.524449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:26.524380    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:24:26.524449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:26.524414    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:24:27.067773 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:27.067741    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" event={"ID":"0cbcf347-7edf-499d-87ea-4ea645f2631e","Type":"ContainerStarted","Data":"7991af225a961b3c7d6afe314eb12a1e06f6d64be83d81aef42d7cc1e2de72d7"}
Oct 31 19:24:27.067951 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:27.067789    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" event={"ID":"0cbcf347-7edf-499d-87ea-4ea645f2631e","Type":"ContainerStarted","Data":"4abd5e356820ab6069039c68988be129aafc67e76e5fe5ed373aca51ba5c17a1"}
Oct 31 19:24:27.241926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:27.241872    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" podStartSLOduration=2.241853991 podStartE2EDuration="2.241853991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:27.238668422 +0000 UTC m=+3813.385118288" watchObservedRunningTime="2026-10-31 19:24:27.241853991 +0000 UTC m=+3813.388303858"
Oct 31 19:24:28.054062 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:28.054033    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:28.055826 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:28.055803    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2gq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:24:28.055952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:28.055844    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 31 19:24:28.085626 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:28.085593    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"]
Oct 31 19:24:28.085951 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:28.085921    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg" podUID="a49c681c-161a-47bd-accc-33a7c1735717" containerName="istio-proxy" containerID="cri-o://59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129" gracePeriod=30
Oct 31 19:24:28.887437 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:28.887396    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:29.053635 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:29.053600    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2gq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:24:29.053812 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:29.053651    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 31 19:24:30.053619 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:30.053588    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2gq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:24:30.054011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:30.053638    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 31 19:24:30.549718 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:30.549685    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:24:31.053717 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:31.053683    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2gq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:24:31.054130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:31.053733    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 31 19:24:32.053906 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:32.053874    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2gq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:24:32.054263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:32.053920    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 31 19:24:33.054016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.053979    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2gq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:24:33.054404 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.054043    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 31 19:24:33.374926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.374898    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:24:33.444764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.444733    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a49c681c-161a-47bd-accc-33a7c1735717-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49c681c-161a-47bd-accc-33a7c1735717-istio-podinfo\") pod \"a49c681c-161a-47bd-accc-33a7c1735717\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") "
Oct 31 19:24:33.444950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.444847    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-certs\") pod \"a49c681c-161a-47bd-accc-33a7c1735717\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") "
Oct 31 19:24:33.444950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.444877    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-data\") pod \"a49c681c-161a-47bd-accc-33a7c1735717\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") "
Oct 31 19:24:33.444950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.444941    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49c681c-161a-47bd-accc-33a7c1735717-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49c681c-161a-47bd-accc-33a7c1735717-istiod-ca-cert\") pod \"a49c681c-161a-47bd-accc-33a7c1735717\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") "
Oct 31 19:24:33.445138 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.444970    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-istio-token\" (UniqueName: \"kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-istio-token\") pod \"a49c681c-161a-47bd-accc-33a7c1735717\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") "
Oct 31 19:24:33.445138 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.445111    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-kube-api-access-st9gw\" (UniqueName: \"kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-kube-api-access-st9gw\") pod \"a49c681c-161a-47bd-accc-33a7c1735717\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") "
Oct 31 19:24:33.445249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.445138    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-envoy\") pod \"a49c681c-161a-47bd-accc-33a7c1735717\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") "
Oct 31 19:24:33.445249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.445170    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-certs" pod "a49c681c-161a-47bd-accc-33a7c1735717" (UID: "a49c681c-161a-47bd-accc-33a7c1735717"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:33.445249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.445194    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-credential-socket\") pod \"a49c681c-161a-47bd-accc-33a7c1735717\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") "
Oct 31 19:24:33.445249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.445232    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-socket\") pod \"a49c681c-161a-47bd-accc-33a7c1735717\" (UID: \"a49c681c-161a-47bd-accc-33a7c1735717\") "
Oct 31 19:24:33.445517 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.445494    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49c681c-161a-47bd-accc-33a7c1735717-istiod-ca-cert" pod "a49c681c-161a-47bd-accc-33a7c1735717" (UID: "a49c681c-161a-47bd-accc-33a7c1735717"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:24:33.445892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.445871    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:33.446034 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.446021    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49c681c-161a-47bd-accc-33a7c1735717-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:33.446125 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.445946    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-socket" pod "a49c681c-161a-47bd-accc-33a7c1735717" (UID: "a49c681c-161a-47bd-accc-33a7c1735717"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:33.446418 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.446400    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-data" pod "a49c681c-161a-47bd-accc-33a7c1735717" (UID: "a49c681c-161a-47bd-accc-33a7c1735717"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:33.446920 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.446900    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-credential-socket" pod "a49c681c-161a-47bd-accc-33a7c1735717" (UID: "a49c681c-161a-47bd-accc-33a7c1735717"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:33.448637 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.448611    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-kube-api-access-st9gw" pod "a49c681c-161a-47bd-accc-33a7c1735717" (UID: "a49c681c-161a-47bd-accc-33a7c1735717"). InnerVolumeSpecName "kube-api-access-st9gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:33.449008 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.448985    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-envoy" pod "a49c681c-161a-47bd-accc-33a7c1735717" (UID: "a49c681c-161a-47bd-accc-33a7c1735717"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:33.455392 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.455368    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-istio-token" pod "a49c681c-161a-47bd-accc-33a7c1735717" (UID: "a49c681c-161a-47bd-accc-33a7c1735717"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:33.462675 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.462394    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a49c681c-161a-47bd-accc-33a7c1735717-istio-podinfo" pod "a49c681c-161a-47bd-accc-33a7c1735717" (UID: "a49c681c-161a-47bd-accc-33a7c1735717"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:24:33.547278 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.547249    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:33.547278 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.547273    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49c681c-161a-47bd-accc-33a7c1735717-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:33.547278 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.547283    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:33.547534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.547291    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:33.547534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.547301    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st9gw\" (UniqueName: \"kubernetes.io/projected/a49c681c-161a-47bd-accc-33a7c1735717-kube-api-access-st9gw\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:33.547534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.547310    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:33.547534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.547318    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49c681c-161a-47bd-accc-33a7c1735717-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:33.557710 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.557671    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:33.730282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:33.730181    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:34.053586 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:34.053556    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2gq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 31 19:24:34.053780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:34.053614    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 31 19:24:34.131062 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:34.131025    4484 generic.go:356] "Generic (PLEG): container finished" podID="a49c681c-161a-47bd-accc-33a7c1735717" containerID="59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129" exitCode=0
Oct 31 19:24:34.131725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:34.131076    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"
Oct 31 19:24:34.131725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:34.131109    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg" event={"ID":"a49c681c-161a-47bd-accc-33a7c1735717","Type":"ContainerDied","Data":"59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129"}
Oct 31 19:24:34.131725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:34.131161    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg" event={"ID":"a49c681c-161a-47bd-accc-33a7c1735717","Type":"ContainerDied","Data":"cb0b4efe7dac3b180e9a7aff2743eede0044f0da8c2e85ffa7653cd04f34ea3d"}
Oct 31 19:24:34.131725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:34.131184    4484 scope.go:122] "RemoveContainer" containerID="59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129"
Oct 31 19:24:34.144048 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:34.144022    4484 scope.go:122] "RemoveContainer" containerID="59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129"
Oct 31 19:24:34.144347 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:24:34.144327    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129\": container with ID starting with 59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129 not found: ID does not exist" containerID="59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129"
Oct 31 19:24:34.144420 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:34.144359    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129"} err="failed to get container status \"59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129\": rpc error: code = NotFound desc = could not find container \"59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129\": container with ID starting with 59c75e5f80440ec35c20fb5ae6fc7be383f348508f0329893db08c96a3d39129 not found: ID does not exist"
Oct 31 19:24:34.302492 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:34.302400    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"]
Oct 31 19:24:34.306678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:34.306651    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1aefcd-openshift-default-85796964fc-v6chg"]
Oct 31 19:24:35.995348 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:35.995249    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49c681c-161a-47bd-accc-33a7c1735717" path="/var/lib/kubelet/pods/a49c681c-161a-47bd-accc-33a7c1735717/volumes"
Oct 31 19:24:36.053850 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:36.053813    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2gq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:24:36.054038 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:36.053864    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:24:37.095688 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:37.095642    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2gq6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:24:37.096191 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:37.095697    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:24:37.261245 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:37.261206    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:37.261769 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:37.261734    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:37.263113 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:37.263091    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:24:38.525664 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:38.525630    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:42.752725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:42.752688    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:44.417143 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:44.417111    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:44.587124 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:44.587086    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:44.872899 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:44.872864    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"]
Oct 31 19:24:44.873138 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:44.873112    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp" podUID="13aca4a2-870e-4b0a-8162-043d154b8702" containerName="istio-proxy" containerID="cri-o://f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384" gracePeriod=30
Oct 31 19:24:44.905791 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:44.905761    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:48.087360 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:48.087314    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:48.254065 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:48.254037    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:48.789770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:48.789736    4484 patch_prober.go:29] interesting pod/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:24:48.789949 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:48.789798    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp" podUID="13aca4a2-870e-4b0a-8162-043d154b8702" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:24:48.889620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:48.889587    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:49.070697 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:49.070612    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:49.458491 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:49.458386    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:50.146214 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.146148    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:24:50.248765 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.248733    4484 generic.go:356] "Generic (PLEG): container finished" podID="13aca4a2-870e-4b0a-8162-043d154b8702" containerID="f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384" exitCode=0
Oct 31 19:24:50.248942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.248782    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"
Oct 31 19:24:50.248942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.248801    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp" event={"ID":"13aca4a2-870e-4b0a-8162-043d154b8702","Type":"ContainerDied","Data":"f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384"}
Oct 31 19:24:50.248942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.248857    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp" event={"ID":"13aca4a2-870e-4b0a-8162-043d154b8702","Type":"ContainerDied","Data":"b18117ee04bb26d29e091c4b7008dccd0a80a35f3b5067e89ba7791db3957489"}
Oct 31 19:24:50.248942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.248887    4484 scope.go:122] "RemoveContainer" containerID="f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384"
Oct 31 19:24:50.254735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.254708    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13aca4a2-870e-4b0a-8162-043d154b8702-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13aca4a2-870e-4b0a-8162-043d154b8702-istiod-ca-cert\") pod \"13aca4a2-870e-4b0a-8162-043d154b8702\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") "
Oct 31 19:24:50.255210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.254778    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13aca4a2-870e-4b0a-8162-043d154b8702-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13aca4a2-870e-4b0a-8162-043d154b8702-istio-podinfo\") pod \"13aca4a2-870e-4b0a-8162-043d154b8702\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") "
Oct 31 19:24:50.255210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.254865    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-socket\") pod \"13aca4a2-870e-4b0a-8162-043d154b8702\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") "
Oct 31 19:24:50.255210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.254895    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-envoy\") pod \"13aca4a2-870e-4b0a-8162-043d154b8702\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") "
Oct 31 19:24:50.255210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.254923    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-istio-token\" (UniqueName: \"kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-istio-token\") pod \"13aca4a2-870e-4b0a-8162-043d154b8702\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") "
Oct 31 19:24:50.255210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.254977    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-data\") pod \"13aca4a2-870e-4b0a-8162-043d154b8702\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") "
Oct 31 19:24:50.255210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.255013    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-credential-socket\") pod \"13aca4a2-870e-4b0a-8162-043d154b8702\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") "
Oct 31 19:24:50.255210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.255152    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-certs\") pod \"13aca4a2-870e-4b0a-8162-043d154b8702\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") "
Oct 31 19:24:50.255210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.255190    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-kube-api-access-n2t54\" (UniqueName: \"kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-kube-api-access-n2t54\") pod \"13aca4a2-870e-4b0a-8162-043d154b8702\" (UID: \"13aca4a2-870e-4b0a-8162-043d154b8702\") "
Oct 31 19:24:50.255684 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.255548    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-data" pod "13aca4a2-870e-4b0a-8162-043d154b8702" (UID: "13aca4a2-870e-4b0a-8162-043d154b8702"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.255769 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.255750    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.256386 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.256359    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13aca4a2-870e-4b0a-8162-043d154b8702-istiod-ca-cert" pod "13aca4a2-870e-4b0a-8162-043d154b8702" (UID: "13aca4a2-870e-4b0a-8162-043d154b8702"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:24:50.256608 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.256585    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-credential-socket" pod "13aca4a2-870e-4b0a-8162-043d154b8702" (UID: "13aca4a2-870e-4b0a-8162-043d154b8702"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.256879 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.256710    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-certs" pod "13aca4a2-870e-4b0a-8162-043d154b8702" (UID: "13aca4a2-870e-4b0a-8162-043d154b8702"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.257053 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.257026    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-socket" pod "13aca4a2-870e-4b0a-8162-043d154b8702" (UID: "13aca4a2-870e-4b0a-8162-043d154b8702"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.258331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.258298    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-kube-api-access-n2t54" pod "13aca4a2-870e-4b0a-8162-043d154b8702" (UID: "13aca4a2-870e-4b0a-8162-043d154b8702"). InnerVolumeSpecName "kube-api-access-n2t54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:50.258810 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.258786    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-envoy" pod "13aca4a2-870e-4b0a-8162-043d154b8702" (UID: "13aca4a2-870e-4b0a-8162-043d154b8702"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:24:50.259132 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.259111    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-istio-token" pod "13aca4a2-870e-4b0a-8162-043d154b8702" (UID: "13aca4a2-870e-4b0a-8162-043d154b8702"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:50.259650 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.259629    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13aca4a2-870e-4b0a-8162-043d154b8702-istio-podinfo" pod "13aca4a2-870e-4b0a-8162-043d154b8702" (UID: "13aca4a2-870e-4b0a-8162-043d154b8702"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:24:50.262070 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.262049    4484 scope.go:122] "RemoveContainer" containerID="f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384"
Oct 31 19:24:50.262406 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:24:50.262383    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384\": container with ID starting with f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384 not found: ID does not exist" containerID="f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384"
Oct 31 19:24:50.262499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.262413    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384"} err="failed to get container status \"f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384\": rpc error: code = NotFound desc = could not find container \"f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384\": container with ID starting with f240e1dbfff8bffbfadaf2e9ca0bb03b2f6b95a739913aaeac981a18600bd384 not found: ID does not exist"
Oct 31 19:24:50.357213 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.357173    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.357213 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.357210    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.357440 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.357223    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2t54\" (UniqueName: \"kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-kube-api-access-n2t54\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.357440 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.357238    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13aca4a2-870e-4b0a-8162-043d154b8702-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.357440 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.357250    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13aca4a2-870e-4b0a-8162-043d154b8702-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.357440 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.357261    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.357440 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.357272    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13aca4a2-870e-4b0a-8162-043d154b8702-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.357440 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.357284    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13aca4a2-870e-4b0a-8162-043d154b8702-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:24:50.683501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.683453    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"]
Oct 31 19:24:50.689029 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:50.688992    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9cf479-openshift-default-75fdb555c-5wqsp"]
Oct 31 19:24:51.994620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:51.994589    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13aca4a2-870e-4b0a-8162-043d154b8702" path="/var/lib/kubelet/pods/13aca4a2-870e-4b0a-8162-043d154b8702/volumes"
Oct 31 19:24:54.382381 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:54.382348    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:54.545404 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:54.545370    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:55.469322 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:24:55.469277    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:24:57.229502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:57.229450    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:57.390159 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:57.390122    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:57.999744 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:57.999717    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:58.876341 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:58.876309    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:59.104245 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:59.104208    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:59.267927 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:59.267894    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:59.774039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:59.774005    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:59.936106 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:24:59.936065    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:01.867317 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:01.867267    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:03.362936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:03.362898    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:03.554760 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:03.554718    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:03.729854 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:03.729774    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:04.262830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:04.262795    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:05.941941 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:05.941905    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:06.123990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:06.123950    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:06.381715 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:06.381686    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:08.492811 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:08.492780    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:08.807289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:08.807207    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:09.064872 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.064794    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"]
Oct 31 19:25:09.065430 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.065417    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a49c681c-161a-47bd-accc-33a7c1735717" containerName="istio-proxy"
Oct 31 19:25:09.065494 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.065432    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49c681c-161a-47bd-accc-33a7c1735717" containerName="istio-proxy"
Oct 31 19:25:09.065494 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.065443    4484 cpu_manager.go:414] "containerMap: removing container" podUID="13aca4a2-870e-4b0a-8162-043d154b8702" containerName="istio-proxy"
Oct 31 19:25:09.065494 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.065449    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13aca4a2-870e-4b0a-8162-043d154b8702" containerName="istio-proxy"
Oct 31 19:25:09.065597 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.065557    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="13aca4a2-870e-4b0a-8162-043d154b8702" containerName="istio-proxy"
Oct 31 19:25:09.065597 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.065576    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49c681c-161a-47bd-accc-33a7c1735717" containerName="istio-proxy"
Oct 31 19:25:09.068984 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.068969    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"
Oct 31 19:25:09.127109 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.127073    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"]
Oct 31 19:25:09.173532 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.173489    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c97ce8-dockercfg-hxxfx\""
Oct 31 19:25:09.173532 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.173520    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c97ce8\""
Oct 31 19:25:09.181704 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.181661    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e5f974-d0c6-4f66-8173-1aafbc948a40-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck\" (UID: \"a6e5f974-d0c6-4f66-8173-1aafbc948a40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"
Oct 31 19:25:09.181858 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.181776    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qk2k\" (UniqueName: \"kubernetes.io/projected/a6e5f974-d0c6-4f66-8173-1aafbc948a40-kube-api-access-2qk2k\") pod \"payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck\" (UID: \"a6e5f974-d0c6-4f66-8173-1aafbc948a40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"
Oct 31 19:25:09.282492 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.282451    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e5f974-d0c6-4f66-8173-1aafbc948a40-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck\" (UID: \"a6e5f974-d0c6-4f66-8173-1aafbc948a40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"
Oct 31 19:25:09.282672 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.282543    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qk2k\" (UniqueName: \"kubernetes.io/projected/a6e5f974-d0c6-4f66-8173-1aafbc948a40-kube-api-access-2qk2k\") pod \"payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck\" (UID: \"a6e5f974-d0c6-4f66-8173-1aafbc948a40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"
Oct 31 19:25:09.283158 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.283138    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e5f974-d0c6-4f66-8173-1aafbc948a40-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck\" (UID: \"a6e5f974-d0c6-4f66-8173-1aafbc948a40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"
Oct 31 19:25:09.341531 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.341445    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qk2k\" (UniqueName: \"kubernetes.io/projected/a6e5f974-d0c6-4f66-8173-1aafbc948a40-kube-api-access-2qk2k\") pod \"payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck\" (UID: \"a6e5f974-d0c6-4f66-8173-1aafbc948a40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"
Oct 31 19:25:09.382711 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.382675    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"
Oct 31 19:25:09.463461 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.463431    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:09.764701 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:09.764673    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"]
Oct 31 19:25:09.765538 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:25:09.765511    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e5f974_d0c6_4f66_8173_1aafbc948a40.slice/crio-7b57e1f94f3be94710c85b24638d4268299d3f05a03eca7087d02ef755e83283 WatchSource:0}: Error finding container 7b57e1f94f3be94710c85b24638d4268299d3f05a03eca7087d02ef755e83283: Status 404 returned error can't find the container with id 7b57e1f94f3be94710c85b24638d4268299d3f05a03eca7087d02ef755e83283
Oct 31 19:25:10.376369 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.376331    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck" event={"ID":"a6e5f974-d0c6-4f66-8173-1aafbc948a40","Type":"ContainerStarted","Data":"a9da4553f9148da24639609d360b9691f3c19e0ea611fe5efa0a6662d537a1af"}
Oct 31 19:25:10.376856 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.376391    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck" event={"ID":"a6e5f974-d0c6-4f66-8173-1aafbc948a40","Type":"ContainerStarted","Data":"7b57e1f94f3be94710c85b24638d4268299d3f05a03eca7087d02ef755e83283"}
Oct 31 19:25:10.376856 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.376527    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"
Oct 31 19:25:10.495694 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.495639    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck" podStartSLOduration=1.495623846 podStartE2EDuration="1.495623846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:10.494447519 +0000 UTC m=+3856.640897383" watchObservedRunningTime="2026-10-31 19:25:10.495623846 +0000 UTC m=+3856.642073709"
Oct 31 19:25:10.684409 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.684331    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"]
Oct 31 19:25:10.688587 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.688572    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.774085 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.774051    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"]
Oct 31 19:25:10.799715 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.799681    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.799872 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.799732    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.799872 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.799837    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfmg\" (UniqueName: \"kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-kube-api-access-bnfmg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.800007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.799899    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.800007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.799930    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.800007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.799966    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.800120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.800095    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d62a86e-a9fd-403b-92c8-b9617f803b85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.800162 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.800122    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.800162 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.800148    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.901433 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.901397    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.901433 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.901438    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.901695 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.901515    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d62a86e-a9fd-403b-92c8-b9617f803b85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.901695 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.901546    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.901695 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.901586    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.901870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.901740    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.901870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.901789    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.901870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.901835    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfmg\" (UniqueName: \"kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-kube-api-access-bnfmg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.902040 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.901876    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.902040 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.901901    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.902040 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.901954    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.902191 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.902066    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.902191 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.902153    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.902663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.902640    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d62a86e-a9fd-403b-92c8-b9617f803b85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.904095 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.904068    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.904293 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.904274    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.961045 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.960970    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:10.961045 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:10.961023    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfmg\" (UniqueName: \"kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-kube-api-access-bnfmg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-cwp95\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:11.001880 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:11.001848    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:11.428764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:11.428736    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"]
Oct 31 19:25:11.429351 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:25:11.429318    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d62a86e_a9fd_403b_92c8_b9617f803b85.slice/crio-44ff0b08c33d13d012290603ff84175f6376dfa31cc11363b04c7eb33f7d0d32 WatchSource:0}: Error finding container 44ff0b08c33d13d012290603ff84175f6376dfa31cc11363b04c7eb33f7d0d32: Status 404 returned error can't find the container with id 44ff0b08c33d13d012290603ff84175f6376dfa31cc11363b04c7eb33f7d0d32
Oct 31 19:25:11.431943 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:11.431904    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:25:11.432065 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:11.431976    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:25:11.432065 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:11.432007    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:25:11.543836 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:11.543783    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:12.208940 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:12.208908    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:12.386500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:12.384019    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:12.394066 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:12.394030    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" event={"ID":"0d62a86e-a9fd-403b-92c8-b9617f803b85","Type":"ContainerStarted","Data":"8b5ce1ad1e47d015773a24cdd8ab9144363f8880ed3d1727b25c3ec79ff45808"}
Oct 31 19:25:12.394254 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:12.394086    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" event={"ID":"0d62a86e-a9fd-403b-92c8-b9617f803b85","Type":"ContainerStarted","Data":"44ff0b08c33d13d012290603ff84175f6376dfa31cc11363b04c7eb33f7d0d32"}
Oct 31 19:25:12.516246 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:12.516185    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" podStartSLOduration=2.5161686850000002 podStartE2EDuration="2.516168685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:12.512255442 +0000 UTC m=+3858.658705306" watchObservedRunningTime="2026-10-31 19:25:12.516168685 +0000 UTC m=+3858.662618599"
Oct 31 19:25:13.002381 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:13.002344    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:13.004056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:13.004035    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cwp95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 19:25:13.004133 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:13.004100    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 19:25:14.002785 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:14.002754    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cwp95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 19:25:14.003188 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:14.002812    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 19:25:15.003101 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:15.003062    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cwp95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 19:25:15.003533 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:15.003110    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 19:25:15.087203 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:15.087172    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:16.003130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:16.003093    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cwp95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 19:25:16.003574 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:16.003144    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 19:25:17.002783 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:17.002740    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cwp95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 19:25:17.002994 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:17.002801    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 19:25:18.002258 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:18.002217    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cwp95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 19:25:18.002756 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:18.002286    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 19:25:18.751981 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:18.751944    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:18.957740 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:18.957708    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:19.003060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:19.002977    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cwp95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 19:25:19.003060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:19.003047    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 19:25:20.007518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:20.007489    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:20.008004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:20.007990    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:20.008651 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:20.008630    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:25:20.211915 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:20.211867    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:20.528727 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:20.528690    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:21.210294 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:21.210254    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:21.386052 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:21.386028    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"
Oct 31 19:25:21.424544 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:21.424500    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:24.380348 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.380315    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"]
Oct 31 19:25:24.380895 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.380575    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb" podUID="7b2d55ac-f4ea-48db-8e9a-42debccb1803" containerName="maas-api" containerID="cri-o://a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4" gracePeriod=30
Oct 31 19:25:24.503119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.503085    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:24.656740 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.656713    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:25:24.711441 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.711414    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-metrics-certs\") pod \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") "
Oct 31 19:25:24.711646 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.711455    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2d55ac-f4ea-48db-8e9a-42debccb1803-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2d55ac-f4ea-48db-8e9a-42debccb1803-openshift-service-ca\") pod \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") "
Oct 31 19:25:24.711646 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.711516    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2d55ac-f4ea-48db-8e9a-42debccb1803-kube-api-access-g7j2m\" (UniqueName: \"kubernetes.io/projected/7b2d55ac-f4ea-48db-8e9a-42debccb1803-kube-api-access-g7j2m\") pod \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") "
Oct 31 19:25:24.711770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.711684    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-maas-api-tls\") pod \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\" (UID: \"7b2d55ac-f4ea-48db-8e9a-42debccb1803\") "
Oct 31 19:25:24.711885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.711861    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2d55ac-f4ea-48db-8e9a-42debccb1803-openshift-service-ca" pod "7b2d55ac-f4ea-48db-8e9a-42debccb1803" (UID: "7b2d55ac-f4ea-48db-8e9a-42debccb1803"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:25:24.712228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.712207    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2d55ac-f4ea-48db-8e9a-42debccb1803-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:24.713742 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.713717    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-metrics-certs" pod "7b2d55ac-f4ea-48db-8e9a-42debccb1803" (UID: "7b2d55ac-f4ea-48db-8e9a-42debccb1803"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:25:24.713848 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.713784    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-maas-api-tls" pod "7b2d55ac-f4ea-48db-8e9a-42debccb1803" (UID: "7b2d55ac-f4ea-48db-8e9a-42debccb1803"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:25:24.713848 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.713824    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2d55ac-f4ea-48db-8e9a-42debccb1803-kube-api-access-g7j2m" pod "7b2d55ac-f4ea-48db-8e9a-42debccb1803" (UID: "7b2d55ac-f4ea-48db-8e9a-42debccb1803"). InnerVolumeSpecName "kube-api-access-g7j2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:25:24.813454 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.813428    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:24.813454 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.813452    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7j2m\" (UniqueName: \"kubernetes.io/projected/7b2d55ac-f4ea-48db-8e9a-42debccb1803-kube-api-access-g7j2m\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:24.813454 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:24.813462    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2d55ac-f4ea-48db-8e9a-42debccb1803-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:25:25.470365 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:25:25.470324    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:25:25.478723 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:25.478684    4484 generic.go:356] "Generic (PLEG): container finished" podID="7b2d55ac-f4ea-48db-8e9a-42debccb1803" containerID="a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4" exitCode=0
Oct 31 19:25:25.478873 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:25.478725    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"
Oct 31 19:25:25.478873 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:25.478720    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb" event={"ID":"7b2d55ac-f4ea-48db-8e9a-42debccb1803","Type":"ContainerDied","Data":"a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4"}
Oct 31 19:25:25.478873 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:25.478796    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb" event={"ID":"7b2d55ac-f4ea-48db-8e9a-42debccb1803","Type":"ContainerDied","Data":"423ab42f4ae893bb6141257715c14f424b347c1ce600e7fd468d82585bee2437"}
Oct 31 19:25:25.478873 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:25.478825    4484 scope.go:122] "RemoveContainer" containerID="a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4"
Oct 31 19:25:25.489349 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:25.489335    4484 scope.go:122] "RemoveContainer" containerID="a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4"
Oct 31 19:25:25.489651 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:25:25.489630    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4\": container with ID starting with a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4 not found: ID does not exist" containerID="a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4"
Oct 31 19:25:25.489724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:25.489657    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4"} err="failed to get container status \"a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4\": rpc error: code = NotFound desc = could not find container \"a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4\": container with ID starting with a86d6b6fae8fe49042a988c56ee2a8c29c0884fc4d9a8d79f3b2f7a92b13b1c4 not found: ID does not exist"
Oct 31 19:25:25.559985 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:25.559945    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:25.654232 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:25.654194    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"]
Oct 31 19:25:25.700855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:25.700818    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3744b0-5b98dcfdb8-w4qfb"]
Oct 31 19:25:25.772567 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:25.772524    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:25.996236 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:25.996196    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2d55ac-f4ea-48db-8e9a-42debccb1803" path="/var/lib/kubelet/pods/7b2d55ac-f4ea-48db-8e9a-42debccb1803/volumes"
Oct 31 19:25:26.350830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:26.350795    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:28.530710 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:28.530676    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:28.701312 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:28.701264    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:28.878938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:28.878842    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:29.048624 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:29.048589    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:30.026973 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:30.026935    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:30.484506 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:30.484380    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:31.321193 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:31.321161    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:31.677389 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:31.677298    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:31.891174 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:31.891143    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:32.145912 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:32.145861    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:33.021210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:33.021178    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:25:35.621100 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:35.621064    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:35.796539 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:35.796507    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:35.994707 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:35.994662    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:36.050635 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:36.050594    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cwp95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:25:36.050819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:36.050696    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:25:36.878299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:36.878261    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:38.817001 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:38.816962    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:38.986154 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:38.986116    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:39.200381 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.200293    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"]
Oct 31 19:25:39.201047 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.201026    4484 cpu_manager.go:414] "containerMap: removing container" podUID="7b2d55ac-f4ea-48db-8e9a-42debccb1803" containerName="maas-api"
Oct 31 19:25:39.201047 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.201047    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2d55ac-f4ea-48db-8e9a-42debccb1803" containerName="maas-api"
Oct 31 19:25:39.201233 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.201206    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2d55ac-f4ea-48db-8e9a-42debccb1803" containerName="maas-api"
Oct 31 19:25:39.204940 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.204919    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"
Oct 31 19:25:39.261062 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.261034    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"]
Oct 31 19:25:39.325499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.297228    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763e40d8-c753-4cef-8970-e9aa0614d081-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7\" (UID: \"763e40d8-c753-4cef-8970-e9aa0614d081\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"
Oct 31 19:25:39.325499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.297319    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2545\" (UniqueName: \"kubernetes.io/projected/763e40d8-c753-4cef-8970-e9aa0614d081-kube-api-access-w2545\") pod \"payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7\" (UID: \"763e40d8-c753-4cef-8970-e9aa0614d081\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"
Oct 31 19:25:39.326025 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.325793    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-227f0f\""
Oct 31 19:25:39.326025 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.325924    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-227f0f-dockercfg-4jpnx\""
Oct 31 19:25:39.379910 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.379878    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"]
Oct 31 19:25:39.384101 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.384084    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"
Oct 31 19:25:39.399092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.399067    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763e40d8-c753-4cef-8970-e9aa0614d081-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7\" (UID: \"763e40d8-c753-4cef-8970-e9aa0614d081\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"
Oct 31 19:25:39.399215 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.399116    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2545\" (UniqueName: \"kubernetes.io/projected/763e40d8-c753-4cef-8970-e9aa0614d081-kube-api-access-w2545\") pod \"payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7\" (UID: \"763e40d8-c753-4cef-8970-e9aa0614d081\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"
Oct 31 19:25:39.399708 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.399692    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763e40d8-c753-4cef-8970-e9aa0614d081-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7\" (UID: \"763e40d8-c753-4cef-8970-e9aa0614d081\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"
Oct 31 19:25:39.412574 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.412553    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"]
Oct 31 19:25:39.457613 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.457542    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2545\" (UniqueName: \"kubernetes.io/projected/763e40d8-c753-4cef-8970-e9aa0614d081-kube-api-access-w2545\") pod \"payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7\" (UID: \"763e40d8-c753-4cef-8970-e9aa0614d081\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"
Oct 31 19:25:39.500681 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.500645    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt\" (UID: \"9af76e82-bdf5-4b61-8e8d-982eb3721ed7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"
Oct 31 19:25:39.501012 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.500989    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcm7r\" (UniqueName: \"kubernetes.io/projected/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-kube-api-access-rcm7r\") pod \"payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt\" (UID: \"9af76e82-bdf5-4b61-8e8d-982eb3721ed7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"
Oct 31 19:25:39.519113 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.519071    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"
Oct 31 19:25:39.602720 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.602683    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt\" (UID: \"9af76e82-bdf5-4b61-8e8d-982eb3721ed7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"
Oct 31 19:25:39.603059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.603034    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcm7r\" (UniqueName: \"kubernetes.io/projected/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-kube-api-access-rcm7r\") pod \"payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt\" (UID: \"9af76e82-bdf5-4b61-8e8d-982eb3721ed7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"
Oct 31 19:25:39.603425 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.603404    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt\" (UID: \"9af76e82-bdf5-4b61-8e8d-982eb3721ed7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"
Oct 31 19:25:39.662093 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.662060    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcm7r\" (UniqueName: \"kubernetes.io/projected/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-kube-api-access-rcm7r\") pod \"payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt\" (UID: \"9af76e82-bdf5-4b61-8e8d-982eb3721ed7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"
Oct 31 19:25:39.697835 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.697807    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"
Oct 31 19:25:39.894791 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:39.894761    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"]
Oct 31 19:25:39.895697 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:25:39.895675    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763e40d8_c753_4cef_8970_e9aa0614d081.slice/crio-cb9ff9f4e2fb42c70136604ae7a54c1e7ba6366f249810f0f51eab05aeff6296 WatchSource:0}: Error finding container cb9ff9f4e2fb42c70136604ae7a54c1e7ba6366f249810f0f51eab05aeff6296: Status 404 returned error can't find the container with id cb9ff9f4e2fb42c70136604ae7a54c1e7ba6366f249810f0f51eab05aeff6296
Oct 31 19:25:40.074096 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:40.074071    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"]
Oct 31 19:25:40.075009 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:25:40.074933    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af76e82_bdf5_4b61_8e8d_982eb3721ed7.slice/crio-7198e1fcad7cd3628f290135b39e54d357bd031c15bbb148338609b73df05660 WatchSource:0}: Error finding container 7198e1fcad7cd3628f290135b39e54d357bd031c15bbb148338609b73df05660: Status 404 returned error can't find the container with id 7198e1fcad7cd3628f290135b39e54d357bd031c15bbb148338609b73df05660
Oct 31 19:25:40.576299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:40.576258    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7" event={"ID":"763e40d8-c753-4cef-8970-e9aa0614d081","Type":"ContainerStarted","Data":"46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe"}
Oct 31 19:25:40.576521 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:40.576467    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"
Oct 31 19:25:40.576584 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:40.576523    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7" event={"ID":"763e40d8-c753-4cef-8970-e9aa0614d081","Type":"ContainerStarted","Data":"cb9ff9f4e2fb42c70136604ae7a54c1e7ba6366f249810f0f51eab05aeff6296"}
Oct 31 19:25:40.577742 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:40.577716    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt" event={"ID":"9af76e82-bdf5-4b61-8e8d-982eb3721ed7","Type":"ContainerStarted","Data":"cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54"}
Oct 31 19:25:40.577824 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:40.577766    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt" event={"ID":"9af76e82-bdf5-4b61-8e8d-982eb3721ed7","Type":"ContainerStarted","Data":"7198e1fcad7cd3628f290135b39e54d357bd031c15bbb148338609b73df05660"}
Oct 31 19:25:40.577824 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:40.577796    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"
Oct 31 19:25:40.695119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:40.695069    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7" podStartSLOduration=1.6950564049999999 podStartE2EDuration="1.695056405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:40.69173542 +0000 UTC m=+3886.838185283" watchObservedRunningTime="2026-10-31 19:25:40.695056405 +0000 UTC m=+3886.841506269"
Oct 31 19:25:40.808827 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:40.808764    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt" podStartSLOduration=1.808749151 podStartE2EDuration="1.808749151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:40.806272612 +0000 UTC m=+3886.952722477" watchObservedRunningTime="2026-10-31 19:25:40.808749151 +0000 UTC m=+3886.955199009"
Oct 31 19:25:40.960946 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:40.960865    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"]
Oct 31 19:25:40.965274 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:40.965254    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.021394 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.021360    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8082f048-2aba-4b78-8f8e-dedf3015ea89-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.021616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.021403    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.021616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.021438    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8082f048-2aba-4b78-8f8e-dedf3015ea89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.021616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.021551    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8082f048-2aba-4b78-8f8e-dedf3015ea89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.021809 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.021724    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbwd8\" (UniqueName: \"kubernetes.io/projected/8082f048-2aba-4b78-8f8e-dedf3015ea89-kube-api-access-kbwd8\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.021809 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.021800    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.021921 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.021836    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.021921 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.021867    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.021921 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.021896    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.084802 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.084772    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"]
Oct 31 19:25:41.123228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.123194    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8082f048-2aba-4b78-8f8e-dedf3015ea89-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.123228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.123233    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.123519 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.123261    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8082f048-2aba-4b78-8f8e-dedf3015ea89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.123519 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.123379    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8082f048-2aba-4b78-8f8e-dedf3015ea89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.123633 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.123616    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwd8\" (UniqueName: \"kubernetes.io/projected/8082f048-2aba-4b78-8f8e-dedf3015ea89-kube-api-access-kbwd8\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.123716 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.123692    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.123776 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.123703    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.123838 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.123778    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.123838 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.123812    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.123956 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.123844    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.124011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.123983    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.124071 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.124049    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.124124 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.124067    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8082f048-2aba-4b78-8f8e-dedf3015ea89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.124181 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.124171    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.126098 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.126077    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8082f048-2aba-4b78-8f8e-dedf3015ea89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.126195 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.126097    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8082f048-2aba-4b78-8f8e-dedf3015ea89-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.181115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.181093    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8082f048-2aba-4b78-8f8e-dedf3015ea89-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.181246 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.181196    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwd8\" (UniqueName: \"kubernetes.io/projected/8082f048-2aba-4b78-8f8e-dedf3015ea89-kube-api-access-kbwd8\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fs5lm\" (UID: \"8082f048-2aba-4b78-8f8e-dedf3015ea89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.280580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.280552    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:41.708225 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.708200    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"]
Oct 31 19:25:41.708372 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:25:41.708314    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8082f048_2aba_4b78_8f8e_dedf3015ea89.slice/crio-3a5f20141eee39d1d119c1738190ea7cecf6824fdb301eb7233086035a422311 WatchSource:0}: Error finding container 3a5f20141eee39d1d119c1738190ea7cecf6824fdb301eb7233086035a422311: Status 404 returned error can't find the container with id 3a5f20141eee39d1d119c1738190ea7cecf6824fdb301eb7233086035a422311
Oct 31 19:25:41.710918 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.710885    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:25:41.710992 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.710948    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:25:41.710992 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:41.710977    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:25:42.143037 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:42.142989    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:42.594852 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:42.594809    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm" event={"ID":"8082f048-2aba-4b78-8f8e-dedf3015ea89","Type":"ContainerStarted","Data":"4be300eec41532a21a31f1c338141f49fb6af8b66f50c6f133821c5a08ea1f43"}
Oct 31 19:25:42.595037 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:42.594874    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm" event={"ID":"8082f048-2aba-4b78-8f8e-dedf3015ea89","Type":"ContainerStarted","Data":"3a5f20141eee39d1d119c1738190ea7cecf6824fdb301eb7233086035a422311"}
Oct 31 19:25:42.716213 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:42.716149    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm" podStartSLOduration=2.716130928 podStartE2EDuration="2.716130928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:42.712864216 +0000 UTC m=+3888.859314081" watchObservedRunningTime="2026-10-31 19:25:42.716130928 +0000 UTC m=+3888.862580791"
Oct 31 19:25:43.285685 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:43.285643    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:43.287498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:43.287453    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fs5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:25:43.287646 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:43.287529    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm" podUID="8082f048-2aba-4b78-8f8e-dedf3015ea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:25:44.281964 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:44.281927    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fs5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:25:44.282225 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:44.282200    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm" podUID="8082f048-2aba-4b78-8f8e-dedf3015ea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:25:45.284498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:45.283858    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fs5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:25:45.284498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:45.283922    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm" podUID="8082f048-2aba-4b78-8f8e-dedf3015ea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:25:46.286252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:46.285811    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fs5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:25:46.286252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:46.285873    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm" podUID="8082f048-2aba-4b78-8f8e-dedf3015ea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:25:47.287543 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:47.284744    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fs5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:25:47.287543 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:47.284807    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm" podUID="8082f048-2aba-4b78-8f8e-dedf3015ea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:25:47.551455 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:47.551359    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:48.281513 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:48.281445    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fs5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:25:48.281722 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:48.281539    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm" podUID="8082f048-2aba-4b78-8f8e-dedf3015ea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:25:49.281546 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:49.281509    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fs5lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:25:49.281960 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:49.281562    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm" podUID="8082f048-2aba-4b78-8f8e-dedf3015ea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:25:50.286135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:50.286103    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:50.286530 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:50.286369    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:50.287085 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:50.287065    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm"
Oct 31 19:25:51.586597 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:51.586573    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"
Oct 31 19:25:51.587008 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:51.586645    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"
Oct 31 19:25:52.298281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:52.298248    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:54.778216 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:54.778188    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:25:54.780667 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:54.780649    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:25:55.469985 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:25:55.469948    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:25:56.966112 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:25:56.966071    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:01.981951 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:01.981918    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:02.214735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:02.214456    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:02.398425 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:02.398392    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:03.690786 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:03.690747    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:03.871219 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:03.871178    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:04.334127 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:04.334029    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:04.629428 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:04.629315    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:06.719915 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:06.719835    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:07.122569 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:07.122537    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:09.705328 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:09.705291    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:09.868519 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:09.868454    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:10.034399 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:10.034371    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:12.291297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:12.291264    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:12.575072 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:12.574978    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:12.832011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:12.831935    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:13.065552 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.065523    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:13.314092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.314057    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:13.427161 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.427113    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"]
Oct 31 19:26:13.438786 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.438756    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"
Oct 31 19:26:13.489824 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.489790    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"]
Oct 31 19:26:13.494403 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.494366    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c249cd\""
Oct 31 19:26:13.535925 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.535893    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c249cd-dockercfg-rlmqm\""
Oct 31 19:26:13.556369 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.556338    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:13.599650 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.599557    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"]
Oct 31 19:26:13.603769 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.603749    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"
Oct 31 19:26:13.635761 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.635728    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cae31d6-d498-4abc-8151-95c59aeead05-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl\" (UID: \"9cae31d6-d498-4abc-8151-95c59aeead05\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"
Oct 31 19:26:13.635921 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.635773    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75wsr\" (UniqueName: \"kubernetes.io/projected/9cae31d6-d498-4abc-8151-95c59aeead05-kube-api-access-75wsr\") pod \"payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl\" (UID: \"9cae31d6-d498-4abc-8151-95c59aeead05\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"
Oct 31 19:26:13.659523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.659498    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"]
Oct 31 19:26:13.737340 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.737310    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6a4ff5-d235-47ff-804c-314cfa54eb17-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj\" (UID: \"ad6a4ff5-d235-47ff-804c-314cfa54eb17\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"
Oct 31 19:26:13.737537 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.737459    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97mc5\" (UniqueName: \"kubernetes.io/projected/ad6a4ff5-d235-47ff-804c-314cfa54eb17-kube-api-access-97mc5\") pod \"payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj\" (UID: \"ad6a4ff5-d235-47ff-804c-314cfa54eb17\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"
Oct 31 19:26:13.737619 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.737600    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cae31d6-d498-4abc-8151-95c59aeead05-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl\" (UID: \"9cae31d6-d498-4abc-8151-95c59aeead05\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"
Oct 31 19:26:13.737657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.737631    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75wsr\" (UniqueName: \"kubernetes.io/projected/9cae31d6-d498-4abc-8151-95c59aeead05-kube-api-access-75wsr\") pod \"payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl\" (UID: \"9cae31d6-d498-4abc-8151-95c59aeead05\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"
Oct 31 19:26:13.738250 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.738233    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cae31d6-d498-4abc-8151-95c59aeead05-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl\" (UID: \"9cae31d6-d498-4abc-8151-95c59aeead05\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"
Oct 31 19:26:13.796408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.796376    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75wsr\" (UniqueName: \"kubernetes.io/projected/9cae31d6-d498-4abc-8151-95c59aeead05-kube-api-access-75wsr\") pod \"payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl\" (UID: \"9cae31d6-d498-4abc-8151-95c59aeead05\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"
Oct 31 19:26:13.801118 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.801092    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:13.838457 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.838419    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97mc5\" (UniqueName: \"kubernetes.io/projected/ad6a4ff5-d235-47ff-804c-314cfa54eb17-kube-api-access-97mc5\") pod \"payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj\" (UID: \"ad6a4ff5-d235-47ff-804c-314cfa54eb17\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"
Oct 31 19:26:13.838688 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.838534    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6a4ff5-d235-47ff-804c-314cfa54eb17-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj\" (UID: \"ad6a4ff5-d235-47ff-804c-314cfa54eb17\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"
Oct 31 19:26:13.839201 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.839180    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6a4ff5-d235-47ff-804c-314cfa54eb17-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj\" (UID: \"ad6a4ff5-d235-47ff-804c-314cfa54eb17\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"
Oct 31 19:26:13.897031 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.896964    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97mc5\" (UniqueName: \"kubernetes.io/projected/ad6a4ff5-d235-47ff-804c-314cfa54eb17-kube-api-access-97mc5\") pod \"payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj\" (UID: \"ad6a4ff5-d235-47ff-804c-314cfa54eb17\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"
Oct 31 19:26:13.917404 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:13.917368    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"
Oct 31 19:26:14.046294 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:14.046263    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:14.061305 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:14.061270    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"
Oct 31 19:26:14.293278 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:26:14.293250    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6a4ff5_d235_47ff_804c_314cfa54eb17.slice/crio-aa9e800afbf6ca75fc49111e300c020f828e290e842069fbfb4276798e45d9cb WatchSource:0}: Error finding container aa9e800afbf6ca75fc49111e300c020f828e290e842069fbfb4276798e45d9cb: Status 404 returned error can't find the container with id aa9e800afbf6ca75fc49111e300c020f828e290e842069fbfb4276798e45d9cb
Oct 31 19:26:14.293651 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:14.293627    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"]
Oct 31 19:26:14.370322 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:14.370293    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:14.438536 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:26:14.438510    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cae31d6_d498_4abc_8151_95c59aeead05.slice/crio-e215de9b4105b04379ee04bb8244087fadbcb169f4794fb3b635254022cd423c WatchSource:0}: Error finding container e215de9b4105b04379ee04bb8244087fadbcb169f4794fb3b635254022cd423c: Status 404 returned error can't find the container with id e215de9b4105b04379ee04bb8244087fadbcb169f4794fb3b635254022cd423c
Oct 31 19:26:14.438956 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:14.438935    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"]
Oct 31 19:26:14.801951 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:14.801912    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl" event={"ID":"9cae31d6-d498-4abc-8151-95c59aeead05","Type":"ContainerStarted","Data":"8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f"}
Oct 31 19:26:14.802123 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:14.801975    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl" event={"ID":"9cae31d6-d498-4abc-8151-95c59aeead05","Type":"ContainerStarted","Data":"e215de9b4105b04379ee04bb8244087fadbcb169f4794fb3b635254022cd423c"}
Oct 31 19:26:14.802217 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:14.802136    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"
Oct 31 19:26:14.803708 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:14.803685    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj" event={"ID":"ad6a4ff5-d235-47ff-804c-314cfa54eb17","Type":"ContainerStarted","Data":"98c770b51f3a9f8c02b94e8bbb20b61adad4af68e8dec0eec94d493cc6b56b00"}
Oct 31 19:26:14.803844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:14.803720    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj" event={"ID":"ad6a4ff5-d235-47ff-804c-314cfa54eb17","Type":"ContainerStarted","Data":"aa9e800afbf6ca75fc49111e300c020f828e290e842069fbfb4276798e45d9cb"}
Oct 31 19:26:14.803844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:14.803780    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"
Oct 31 19:26:14.917914 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:14.917858    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl" podStartSLOduration=1.9178449610000001 podStartE2EDuration="1.917844961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:26:14.916741698 +0000 UTC m=+3921.063191563" watchObservedRunningTime="2026-10-31 19:26:14.917844961 +0000 UTC m=+3921.064294824"
Oct 31 19:26:15.035009 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:15.034948    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj" podStartSLOduration=2.034929189 podStartE2EDuration="2.034929189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:26:15.031797913 +0000 UTC m=+3921.178247778" watchObservedRunningTime="2026-10-31 19:26:15.034929189 +0000 UTC m=+3921.181379054"
Oct 31 19:26:15.873488 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:15.873453    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:17.450075 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:17.450034    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:18.670893 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:18.670857    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:22.015135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:22.015094    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:22.290845 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:22.290753    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:22.548987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:22.548900    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:23.154127 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:23.154085    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:25.468372 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:26:25.468339    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:26:25.813192 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:25.813145    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"
Oct 31 19:26:25.813460 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:25.813268    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"
Oct 31 19:26:26.930031 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:26.929990    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:29.169622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:29.169581    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:31.677818 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:31.677784    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:36.319768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:36.319724    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:39.834020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:39.833977    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:26:40.927865 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:40.927824    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:45.491178 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:45.491140    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:45.735840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:45.735789    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:50.364874 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:50.364840    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:55.062374 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:55.062334    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:55.343035 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:55.342942    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:55.468770 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:26:55.468736    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:26:59.684535 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:59.684497    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:59.930863 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:26:59.930829    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:02.525685 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:02.525637    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:04.626983 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:04.626936    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:07.017012 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:07.016975    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:09.427985 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:09.427955    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:14.086566 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:14.086528    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:18.785581 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:18.785546    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:23.527382 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:23.527343    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:25.469447 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:27:25.469408    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:27:28.182212 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:28.182171    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:32.949251 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:32.949211    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:35.203506 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:35.203457    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:27:37.583915 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:37.583840    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:37.832914 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:37.832877    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:38.064184 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:38.064152    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:42.490225 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:42.490187    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:47.230682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:47.230648    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:51.927812 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:51.927779    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:55.469217 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:27:55.469177    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:27:56.727418 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:56.727385    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:57.256954 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:57.256918    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:57.719939 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:57.719854    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:58.619715 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:58.619687    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:58.916207 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:27:58.916091    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:01.727910 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:01.727867    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:02.847731 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:02.847663    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:03.189232 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:03.189147    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:03.511793 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:03.511742    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:06.386531 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.386498    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:06.622608 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.622576    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:06.819449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.819419    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"]
Oct 31 19:28:06.823987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.823963    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"
Oct 31 19:28:06.860643 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.860602    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:06.877269 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.877251    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fdd317\""
Oct 31 19:28:06.878065 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.878045    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdgw\" (UniqueName: \"kubernetes.io/projected/c63845cf-8d2c-4bc4-99e0-6576f10f924b-kube-api-access-9rdgw\") pod \"payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww\" (UID: \"c63845cf-8d2c-4bc4-99e0-6576f10f924b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"
Oct 31 19:28:06.878120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.878110    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63845cf-8d2c-4bc4-99e0-6576f10f924b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww\" (UID: \"c63845cf-8d2c-4bc4-99e0-6576f10f924b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"
Oct 31 19:28:06.943530 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.932685    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fdd317-dockercfg-4tj6m\""
Oct 31 19:28:06.971825 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.971252    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"]
Oct 31 19:28:06.978001 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.977970    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"]
Oct 31 19:28:06.980098 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.980063    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdgw\" (UniqueName: \"kubernetes.io/projected/c63845cf-8d2c-4bc4-99e0-6576f10f924b-kube-api-access-9rdgw\") pod \"payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww\" (UID: \"c63845cf-8d2c-4bc4-99e0-6576f10f924b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"
Oct 31 19:28:06.980287 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.980165    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63845cf-8d2c-4bc4-99e0-6576f10f924b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww\" (UID: \"c63845cf-8d2c-4bc4-99e0-6576f10f924b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"
Oct 31 19:28:06.980860 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.980841    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63845cf-8d2c-4bc4-99e0-6576f10f924b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww\" (UID: \"c63845cf-8d2c-4bc4-99e0-6576f10f924b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"
Oct 31 19:28:06.982662 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:06.982644    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"
Oct 31 19:28:07.030688 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.030656    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"]
Oct 31 19:28:07.041538 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.041510    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdgw\" (UniqueName: \"kubernetes.io/projected/c63845cf-8d2c-4bc4-99e0-6576f10f924b-kube-api-access-9rdgw\") pod \"payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww\" (UID: \"c63845cf-8d2c-4bc4-99e0-6576f10f924b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"
Oct 31 19:28:07.082031 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.081949    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dctng\" (UniqueName: \"kubernetes.io/projected/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-kube-api-access-dctng\") pod \"payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk\" (UID: \"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"
Oct 31 19:28:07.082180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.082112    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk\" (UID: \"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"
Oct 31 19:28:07.137232 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.137207    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"
Oct 31 19:28:07.182977 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.182936    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctng\" (UniqueName: \"kubernetes.io/projected/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-kube-api-access-dctng\") pod \"payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk\" (UID: \"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"
Oct 31 19:28:07.183164 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.183027    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk\" (UID: \"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"
Oct 31 19:28:07.183858 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.183835    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk\" (UID: \"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"
Oct 31 19:28:07.244267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.244234    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctng\" (UniqueName: \"kubernetes.io/projected/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-kube-api-access-dctng\") pod \"payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk\" (UID: \"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"
Oct 31 19:28:07.296692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.296663    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"
Oct 31 19:28:07.367440 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.367407    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:07.515253 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:28:07.515219    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63845cf_8d2c_4bc4_99e0_6576f10f924b.slice/crio-7fb998941beb22b5cc5d88ea2dad57411447a302830920efa727a6872ae6cb3f WatchSource:0}: Error finding container 7fb998941beb22b5cc5d88ea2dad57411447a302830920efa727a6872ae6cb3f: Status 404 returned error can't find the container with id 7fb998941beb22b5cc5d88ea2dad57411447a302830920efa727a6872ae6cb3f
Oct 31 19:28:07.515635 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.515457    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"]
Oct 31 19:28:07.538397 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.538369    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww" event={"ID":"c63845cf-8d2c-4bc4-99e0-6576f10f924b","Type":"ContainerStarted","Data":"7fb998941beb22b5cc5d88ea2dad57411447a302830920efa727a6872ae6cb3f"}
Oct 31 19:28:07.670771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:07.670741    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"]
Oct 31 19:28:07.671450 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:28:07.671422    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4bd02a_689d_4f80_b2ff_1045c62dfa8a.slice/crio-12fe2f6c93fb2ba0b4f153e9fda8501ca54578edff2232be79657b28a97aca04 WatchSource:0}: Error finding container 12fe2f6c93fb2ba0b4f153e9fda8501ca54578edff2232be79657b28a97aca04: Status 404 returned error can't find the container with id 12fe2f6c93fb2ba0b4f153e9fda8501ca54578edff2232be79657b28a97aca04
Oct 31 19:28:08.546434 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:08.546380    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww" event={"ID":"c63845cf-8d2c-4bc4-99e0-6576f10f924b","Type":"ContainerStarted","Data":"12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7"}
Oct 31 19:28:08.546938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:08.546618    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"
Oct 31 19:28:08.548250 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:08.548225    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk" event={"ID":"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a","Type":"ContainerStarted","Data":"73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd"}
Oct 31 19:28:08.548393 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:08.548267    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk" event={"ID":"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a","Type":"ContainerStarted","Data":"12fe2f6c93fb2ba0b4f153e9fda8501ca54578edff2232be79657b28a97aca04"}
Oct 31 19:28:08.548393 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:08.548300    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"
Oct 31 19:28:08.666194 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:08.666147    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww" podStartSLOduration=2.666134264 podStartE2EDuration="2.666134264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:28:08.663308268 +0000 UTC m=+4034.809758166" watchObservedRunningTime="2026-10-31 19:28:08.666134264 +0000 UTC m=+4034.812584128"
Oct 31 19:28:08.783750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:08.783697    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk" podStartSLOduration=2.783678516 podStartE2EDuration="2.783678516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:28:08.779867432 +0000 UTC m=+4034.926317296" watchObservedRunningTime="2026-10-31 19:28:08.783678516 +0000 UTC m=+4034.930128380"
Oct 31 19:28:11.490902 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:11.490865    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:16.142962 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:16.142927    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:19.558095 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:19.558067    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"
Oct 31 19:28:19.558690 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:19.558137    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"
Oct 31 19:28:20.847209 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:20.847178    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:24.539108 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:24.539072    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:25.467845 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:28:25.467804    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:28:25.830386 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:25.830336    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:28.105031 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:28.104997    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:28:30.384392 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:30.384355    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:30.668916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:30.668835    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:31.627057 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:31.627021    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:31.861064 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:31.861028    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:35.558950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:35.558916    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:35.989854 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:35.989821    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:39.454743 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:39.454704    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:40.653061 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:40.653029    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:45.331346 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:45.331311    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:45.567857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:45.567829    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:46.102772 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:46.102735    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"]
Oct 31 19:28:46.208887 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:46.208847    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"]
Oct 31 19:28:46.448755 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:46.448669    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:46.688994 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:46.688964    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:50.382360 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:50.382320    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:51.010225 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.010184    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:51.103365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.103329    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt" podUID="9af76e82-bdf5-4b61-8e8d-982eb3721ed7" containerName="payload-processing" containerID="cri-o://cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54" gracePeriod=25
Oct 31 19:28:51.209822 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.209787    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7" podUID="763e40d8-c753-4cef-8970-e9aa0614d081" containerName="payload-pre-processing" containerID="cri-o://46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe" gracePeriod=25
Oct 31 19:28:51.380982 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.380954    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"
Oct 31 19:28:51.477534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.477455    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"
Oct 31 19:28:51.517236 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.517208    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-plugins-config-volume\") pod \"9af76e82-bdf5-4b61-8e8d-982eb3721ed7\" (UID: \"9af76e82-bdf5-4b61-8e8d-982eb3721ed7\") "
Oct 31 19:28:51.517515 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.517495    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-kube-api-access-rcm7r\" (UniqueName: \"kubernetes.io/projected/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-kube-api-access-rcm7r\") pod \"9af76e82-bdf5-4b61-8e8d-982eb3721ed7\" (UID: \"9af76e82-bdf5-4b61-8e8d-982eb3721ed7\") "
Oct 31 19:28:51.517653 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.517632    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-plugins-config-volume" pod "9af76e82-bdf5-4b61-8e8d-982eb3721ed7" (UID: "9af76e82-bdf5-4b61-8e8d-982eb3721ed7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:28:51.517930 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.517914    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:28:51.519759 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.519734    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-kube-api-access-rcm7r" pod "9af76e82-bdf5-4b61-8e8d-982eb3721ed7" (UID: "9af76e82-bdf5-4b61-8e8d-982eb3721ed7"). InnerVolumeSpecName "kube-api-access-rcm7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:28:51.618870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.618787    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763e40d8-c753-4cef-8970-e9aa0614d081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763e40d8-c753-4cef-8970-e9aa0614d081-plugins-config-volume\") pod \"763e40d8-c753-4cef-8970-e9aa0614d081\" (UID: \"763e40d8-c753-4cef-8970-e9aa0614d081\") "
Oct 31 19:28:51.619016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.618885    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763e40d8-c753-4cef-8970-e9aa0614d081-kube-api-access-w2545\" (UniqueName: \"kubernetes.io/projected/763e40d8-c753-4cef-8970-e9aa0614d081-kube-api-access-w2545\") pod \"763e40d8-c753-4cef-8970-e9aa0614d081\" (UID: \"763e40d8-c753-4cef-8970-e9aa0614d081\") "
Oct 31 19:28:51.619244 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.619209    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763e40d8-c753-4cef-8970-e9aa0614d081-plugins-config-volume" pod "763e40d8-c753-4cef-8970-e9aa0614d081" (UID: "763e40d8-c753-4cef-8970-e9aa0614d081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:28:51.619575 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.619555    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcm7r\" (UniqueName: \"kubernetes.io/projected/9af76e82-bdf5-4b61-8e8d-982eb3721ed7-kube-api-access-rcm7r\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:28:51.619643 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.619580    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763e40d8-c753-4cef-8970-e9aa0614d081-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:28:51.621636 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.621607    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763e40d8-c753-4cef-8970-e9aa0614d081-kube-api-access-w2545" pod "763e40d8-c753-4cef-8970-e9aa0614d081" (UID: "763e40d8-c753-4cef-8970-e9aa0614d081"). InnerVolumeSpecName "kube-api-access-w2545". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:28:51.721051 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.721014    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2545\" (UniqueName: \"kubernetes.io/projected/763e40d8-c753-4cef-8970-e9aa0614d081-kube-api-access-w2545\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:28:51.844023 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.843985    4484 generic.go:356] "Generic (PLEG): container finished" podID="9af76e82-bdf5-4b61-8e8d-982eb3721ed7" containerID="cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54" exitCode=0
Oct 31 19:28:51.844211 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.844039    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"
Oct 31 19:28:51.844211 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.844072    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt" event={"ID":"9af76e82-bdf5-4b61-8e8d-982eb3721ed7","Type":"ContainerDied","Data":"cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54"}
Oct 31 19:28:51.844211 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.844135    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt" event={"ID":"9af76e82-bdf5-4b61-8e8d-982eb3721ed7","Type":"ContainerDied","Data":"7198e1fcad7cd3628f290135b39e54d357bd031c15bbb148338609b73df05660"}
Oct 31 19:28:51.844211 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.844158    4484 scope.go:122] "RemoveContainer" containerID="cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54"
Oct 31 19:28:51.845746 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.845723    4484 generic.go:356] "Generic (PLEG): container finished" podID="763e40d8-c753-4cef-8970-e9aa0614d081" containerID="46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe" exitCode=0
Oct 31 19:28:51.845881 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.845771    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"
Oct 31 19:28:51.845881 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.845790    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7" event={"ID":"763e40d8-c753-4cef-8970-e9aa0614d081","Type":"ContainerDied","Data":"46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe"}
Oct 31 19:28:51.845881 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.845829    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7" event={"ID":"763e40d8-c753-4cef-8970-e9aa0614d081","Type":"ContainerDied","Data":"cb9ff9f4e2fb42c70136604ae7a54c1e7ba6366f249810f0f51eab05aeff6296"}
Oct 31 19:28:51.856596 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.856574    4484 scope.go:122] "RemoveContainer" containerID="cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54"
Oct 31 19:28:51.856928 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:28:51.856908    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54\": container with ID starting with cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54 not found: ID does not exist" containerID="cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54"
Oct 31 19:28:51.856994 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.856938    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54"} err="failed to get container status \"cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54\": rpc error: code = NotFound desc = could not find container \"cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54\": container with ID starting with cbbced5e64ceaf14af3d4a42a147838bfab6f526db4821c1ef5c291c1533cd54 not found: ID does not exist"
Oct 31 19:28:51.856994 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.856954    4484 scope.go:122] "RemoveContainer" containerID="46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe"
Oct 31 19:28:51.866293 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.866273    4484 scope.go:122] "RemoveContainer" containerID="46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe"
Oct 31 19:28:51.866626 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:28:51.866607    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe\": container with ID starting with 46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe not found: ID does not exist" containerID="46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe"
Oct 31 19:28:51.866694 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:51.866633    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe"} err="failed to get container status \"46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe\": rpc error: code = NotFound desc = could not find container \"46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe\": container with ID starting with 46209be6a5f3c63f848836811a5854eb7a5e054818bc404c92ed32eeb3570afe not found: ID does not exist"
Oct 31 19:28:52.012589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:52.012560    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"]
Oct 31 19:28:52.063189 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:52.063153    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-227f0f-6458fd6975-vphz7"]
Oct 31 19:28:52.126947 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:52.126924    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"]
Oct 31 19:28:52.132446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:52.132424    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-227f0f-56bd8c965f-djxmt"]
Oct 31 19:28:53.995554 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:53.995525    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763e40d8-c753-4cef-8970-e9aa0614d081" path="/var/lib/kubelet/pods/763e40d8-c753-4cef-8970-e9aa0614d081/volumes"
Oct 31 19:28:53.995946 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:53.995933    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af76e82-bdf5-4b61-8e8d-982eb3721ed7" path="/var/lib/kubelet/pods/9af76e82-bdf5-4b61-8e8d-982eb3721ed7/volumes"
Oct 31 19:28:55.026137 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:55.026103    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:55.468436 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:28:55.468353    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:28:59.740906 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:28:59.740859    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:03.182156 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:03.182110    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:04.417615 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:04.417576    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:07.696062 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:07.695972    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:08.127187 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:08.127154    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:09.070930 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:09.070897    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:13.777986 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:13.777949    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:14.227411 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:14.227334    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:18.740719 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:18.740682    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:19.725826 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:19.725783    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:20.020294 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:20.020264    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:23.518591 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:23.518552    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:25.469655 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:29:25.469620    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:29:28.205636 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:28.205600    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:29.008079 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:29.008042    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:32.791842 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:32.791802    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:33.031220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:33.031183    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:33.268112 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:33.268065    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:37.685187 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:37.685148    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:42.409119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:42.409086    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:46.650227 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.650190    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"]
Oct 31 19:29:46.650893 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.650877    4484 cpu_manager.go:414] "containerMap: removing container" podUID="9af76e82-bdf5-4b61-8e8d-982eb3721ed7" containerName="payload-processing"
Oct 31 19:29:46.650942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.650894    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af76e82-bdf5-4b61-8e8d-982eb3721ed7" containerName="payload-processing"
Oct 31 19:29:46.650942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.650916    4484 cpu_manager.go:414] "containerMap: removing container" podUID="763e40d8-c753-4cef-8970-e9aa0614d081" containerName="payload-pre-processing"
Oct 31 19:29:46.650942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.650921    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763e40d8-c753-4cef-8970-e9aa0614d081" containerName="payload-pre-processing"
Oct 31 19:29:46.651036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.651028    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af76e82-bdf5-4b61-8e8d-982eb3721ed7" containerName="payload-processing"
Oct 31 19:29:46.651069 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.651037    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="763e40d8-c753-4cef-8970-e9aa0614d081" containerName="payload-pre-processing"
Oct 31 19:29:46.654362 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.654347    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:46.683462 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.683433    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec5aa6ba-97vfb\" (UID: \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:46.683618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.683560    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec5aa6ba-97vfb\" (UID: \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:46.683618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.683606    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2nwf\" (UniqueName: \"kubernetes.io/projected/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-kube-api-access-d2nwf\") pod \"maas-api-revoke-sub-ec5aa6ba-97vfb\" (UID: \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:46.710017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.709981    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"]
Oct 31 19:29:46.756878 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.756852    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:29:46.757014 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.756956    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:29:46.784751 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.784726    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec5aa6ba-97vfb\" (UID: \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:46.784876 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.784774    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec5aa6ba-97vfb\" (UID: \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:46.784917 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.784890    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2nwf\" (UniqueName: \"kubernetes.io/projected/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-kube-api-access-d2nwf\") pod \"maas-api-revoke-sub-ec5aa6ba-97vfb\" (UID: \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:46.785434 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.785418    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec5aa6ba-97vfb\" (UID: \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:46.787135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.787120    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec5aa6ba-97vfb\" (UID: \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:46.843540 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.843514    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2nwf\" (UniqueName: \"kubernetes.io/projected/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-kube-api-access-d2nwf\") pod \"maas-api-revoke-sub-ec5aa6ba-97vfb\" (UID: \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:46.968329 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:46.968259    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:47.073369 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:47.073320    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:47.344175 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:29:47.344148    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbf6cd1_0704_4ecd_a16b_7e1d6230745a.slice/crio-0897de6746f8d74f5613aba6e1b05c4184dd3fcd26a368db2e02f6fad1aa8c11 WatchSource:0}: Error finding container 0897de6746f8d74f5613aba6e1b05c4184dd3fcd26a368db2e02f6fad1aa8c11: Status 404 returned error can't find the container with id 0897de6746f8d74f5613aba6e1b05c4184dd3fcd26a368db2e02f6fad1aa8c11
Oct 31 19:29:47.344175 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:47.344163    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"]
Oct 31 19:29:48.215374 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:48.215336    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb" event={"ID":"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a","Type":"ContainerStarted","Data":"932a90506a4ba929cefd50b82e1cbeb349a028bf5539f788ff38fe32a8cdbaa7"}
Oct 31 19:29:48.215797 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:48.215432    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb" event={"ID":"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a","Type":"ContainerStarted","Data":"0897de6746f8d74f5613aba6e1b05c4184dd3fcd26a368db2e02f6fad1aa8c11"}
Oct 31 19:29:48.336281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:48.336229    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb" podStartSLOduration=2.336215521 podStartE2EDuration="2.336215521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:48.333270784 +0000 UTC m=+4134.479720648" watchObservedRunningTime="2026-10-31 19:29:48.336215521 +0000 UTC m=+4134.482665381"
Oct 31 19:29:49.223461 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:49.223429    4484 generic.go:356] "Generic (PLEG): container finished" podID="fdbf6cd1-0704-4ecd-a16b-7e1d6230745a" containerID="932a90506a4ba929cefd50b82e1cbeb349a028bf5539f788ff38fe32a8cdbaa7" exitCode=0
Oct 31 19:29:49.223866 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:49.223518    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb" event={"ID":"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a","Type":"ContainerDied","Data":"932a90506a4ba929cefd50b82e1cbeb349a028bf5539f788ff38fe32a8cdbaa7"}
Oct 31 19:29:50.422840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:50.422817    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:50.529722 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:50.529693    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-openshift-ca\") pod \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\" (UID: \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\") "
Oct 31 19:29:50.529895 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:50.529759    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-xks-ca\") pod \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\" (UID: \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\") "
Oct 31 19:29:50.529895 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:50.529842    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-kube-api-access-d2nwf\" (UniqueName: \"kubernetes.io/projected/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-kube-api-access-d2nwf\") pod \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\" (UID: \"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a\") "
Oct 31 19:29:50.530022 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:50.530002    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-openshift-ca" pod "fdbf6cd1-0704-4ecd-a16b-7e1d6230745a" (UID: "fdbf6cd1-0704-4ecd-a16b-7e1d6230745a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:29:50.530188 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:50.530174    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:29:50.531933 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:50.531904    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-xks-ca" pod "fdbf6cd1-0704-4ecd-a16b-7e1d6230745a" (UID: "fdbf6cd1-0704-4ecd-a16b-7e1d6230745a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:29:50.532054 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:50.532035    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-kube-api-access-d2nwf" pod "fdbf6cd1-0704-4ecd-a16b-7e1d6230745a" (UID: "fdbf6cd1-0704-4ecd-a16b-7e1d6230745a"). InnerVolumeSpecName "kube-api-access-d2nwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:29:50.612341 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:50.612308    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:29:50.631156 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:50.631132    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2nwf\" (UniqueName: \"kubernetes.io/projected/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-kube-api-access-d2nwf\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:29:50.631156 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:50.631156    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:29:51.171996 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:51.171918    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:51.237168 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:51.237137    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"
Oct 31 19:29:51.237356 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:51.237173    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb" event={"ID":"fdbf6cd1-0704-4ecd-a16b-7e1d6230745a","Type":"ContainerDied","Data":"0897de6746f8d74f5613aba6e1b05c4184dd3fcd26a368db2e02f6fad1aa8c11"}
Oct 31 19:29:51.237356 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:51.237221    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0897de6746f8d74f5613aba6e1b05c4184dd3fcd26a368db2e02f6fad1aa8c11"
Oct 31 19:29:52.893422 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:52.893389    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:55.471143 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:29:55.471104    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:29:56.045695 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:56.045660    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:56.677432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:56.677395    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:57.785566 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:57.785514    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:59.389653 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:59.389616    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:59.648148 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:59.648055    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:59.965699 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:29:59.965586    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:00.177323 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.177282    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"]
Oct 31 19:30:00.178054 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.178036    4484 cpu_manager.go:414] "containerMap: removing container" podUID="fdbf6cd1-0704-4ecd-a16b-7e1d6230745a" containerName="revoke-keys"
Oct 31 19:30:00.178125 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.178055    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbf6cd1-0704-4ecd-a16b-7e1d6230745a" containerName="revoke-keys"
Oct 31 19:30:00.178220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.178210    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbf6cd1-0704-4ecd-a16b-7e1d6230745a" containerName="revoke-keys"
Oct 31 19:30:00.182705 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.182684    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"]
Oct 31 19:30:00.182848 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.182803    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"
Oct 31 19:30:00.186850 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.186820    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"]
Oct 31 19:30:00.186997 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.186897    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"
Oct 31 19:30:00.190836 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.190815    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"
Oct 31 19:30:00.241584 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.237796    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"]
Oct 31 19:30:00.241584 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.241355    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:30:00.245498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.245023    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"
Oct 31 19:30:00.257564 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.257539    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"]
Oct 31 19:30:00.276218 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.273568    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5467e676-3261-4bb9-9ef4-b8a296072512-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v\" (UID: \"5467e676-3261-4bb9-9ef4-b8a296072512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"
Oct 31 19:30:00.279589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.279565    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lxk5\" (UniqueName: \"kubernetes.io/projected/5467e676-3261-4bb9-9ef4-b8a296072512-kube-api-access-6lxk5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v\" (UID: \"5467e676-3261-4bb9-9ef4-b8a296072512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"
Oct 31 19:30:00.283300 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.281917    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"
Oct 31 19:30:00.335269 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.318519    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"]
Oct 31 19:30:00.381432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.381400    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lxk5\" (UniqueName: \"kubernetes.io/projected/5467e676-3261-4bb9-9ef4-b8a296072512-kube-api-access-6lxk5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v\" (UID: \"5467e676-3261-4bb9-9ef4-b8a296072512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"
Oct 31 19:30:00.385502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.385457    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"]
Oct 31 19:30:00.386733 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.386707    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"
Oct 31 19:30:00.387100 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.387075    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6\" (UID: \"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"
Oct 31 19:30:00.387198 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.387119    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwkvt\" (UniqueName: \"kubernetes.io/projected/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-kube-api-access-xwkvt\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6\" (UID: \"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"
Oct 31 19:30:00.387263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.387210    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f886bc5-ea49-4717-8e09-d3fe02c3d334-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp\" (UID: \"0f886bc5-ea49-4717-8e09-d3fe02c3d334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"
Oct 31 19:30:00.387320 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.387265    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwg2\" (UniqueName: \"kubernetes.io/projected/ba38b8bd-6dcb-4b68-9226-2455456bcf74-kube-api-access-rfwg2\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p\" (UID: \"ba38b8bd-6dcb-4b68-9226-2455456bcf74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"
Oct 31 19:30:00.387376 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.387342    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5467e676-3261-4bb9-9ef4-b8a296072512-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v\" (UID: \"5467e676-3261-4bb9-9ef4-b8a296072512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"
Oct 31 19:30:00.387432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.387407    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73492d9-5852-498a-a6b7-10e2f446feeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84\" (UID: \"f73492d9-5852-498a-a6b7-10e2f446feeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"
Oct 31 19:30:00.387509 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.387442    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fms6\" (UniqueName: \"kubernetes.io/projected/0f886bc5-ea49-4717-8e09-d3fe02c3d334-kube-api-access-4fms6\") pod \"maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp\" (UID: \"0f886bc5-ea49-4717-8e09-d3fe02c3d334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"
Oct 31 19:30:00.387698 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.387525    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba38b8bd-6dcb-4b68-9226-2455456bcf74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p\" (UID: \"ba38b8bd-6dcb-4b68-9226-2455456bcf74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"
Oct 31 19:30:00.387698 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.387666    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9h9j\" (UniqueName: \"kubernetes.io/projected/f73492d9-5852-498a-a6b7-10e2f446feeb-kube-api-access-f9h9j\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84\" (UID: \"f73492d9-5852-498a-a6b7-10e2f446feeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"
Oct 31 19:30:00.389291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.389263    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5467e676-3261-4bb9-9ef4-b8a296072512-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v\" (UID: \"5467e676-3261-4bb9-9ef4-b8a296072512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"
Oct 31 19:30:00.397542 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.397346    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"]
Oct 31 19:30:00.397542 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.397396    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"]
Oct 31 19:30:00.398261 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.398239    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"
Oct 31 19:30:00.407768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.407711    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"]
Oct 31 19:30:00.408024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.408008    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"]
Oct 31 19:30:00.408248 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.408236    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"]
Oct 31 19:30:00.408846 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.408157    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"
Oct 31 19:30:00.416510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.413354    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"]
Oct 31 19:30:00.416898 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.416876    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"]
Oct 31 19:30:00.424854 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.424813    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"]
Oct 31 19:30:00.428152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.428134    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"]
Oct 31 19:30:00.444817 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.444796    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lxk5\" (UniqueName: \"kubernetes.io/projected/5467e676-3261-4bb9-9ef4-b8a296072512-kube-api-access-6lxk5\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v\" (UID: \"5467e676-3261-4bb9-9ef4-b8a296072512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"
Oct 31 19:30:00.489114 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.489087    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba38b8bd-6dcb-4b68-9226-2455456bcf74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p\" (UID: \"ba38b8bd-6dcb-4b68-9226-2455456bcf74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"
Oct 31 19:30:00.489235 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.489138    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9h9j\" (UniqueName: \"kubernetes.io/projected/f73492d9-5852-498a-a6b7-10e2f446feeb-kube-api-access-f9h9j\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84\" (UID: \"f73492d9-5852-498a-a6b7-10e2f446feeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"
Oct 31 19:30:00.489347 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.489329    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6mh\" (UniqueName: \"kubernetes.io/projected/a901ad86-48e1-422b-a342-64e3433fd101-kube-api-access-xk6mh\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb\" (UID: \"a901ad86-48e1-422b-a342-64e3433fd101\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"
Oct 31 19:30:00.489591 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.489572    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a901ad86-48e1-422b-a342-64e3433fd101-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb\" (UID: \"a901ad86-48e1-422b-a342-64e3433fd101\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"
Oct 31 19:30:00.489746 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.489728    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6\" (UID: \"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"
Oct 31 19:30:00.489812 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.489759    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwkvt\" (UniqueName: \"kubernetes.io/projected/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-kube-api-access-xwkvt\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6\" (UID: \"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"
Oct 31 19:30:00.489866 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.489813    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f886bc5-ea49-4717-8e09-d3fe02c3d334-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp\" (UID: \"0f886bc5-ea49-4717-8e09-d3fe02c3d334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"
Oct 31 19:30:00.489866 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.489850    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwg2\" (UniqueName: \"kubernetes.io/projected/ba38b8bd-6dcb-4b68-9226-2455456bcf74-kube-api-access-rfwg2\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p\" (UID: \"ba38b8bd-6dcb-4b68-9226-2455456bcf74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"
Oct 31 19:30:00.489866 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.489852    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba38b8bd-6dcb-4b68-9226-2455456bcf74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p\" (UID: \"ba38b8bd-6dcb-4b68-9226-2455456bcf74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"
Oct 31 19:30:00.490043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.489961    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73492d9-5852-498a-a6b7-10e2f446feeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84\" (UID: \"f73492d9-5852-498a-a6b7-10e2f446feeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"
Oct 31 19:30:00.490043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.489999    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fms6\" (UniqueName: \"kubernetes.io/projected/0f886bc5-ea49-4717-8e09-d3fe02c3d334-kube-api-access-4fms6\") pod \"maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp\" (UID: \"0f886bc5-ea49-4717-8e09-d3fe02c3d334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"
Oct 31 19:30:00.490380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.490357    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6\" (UID: \"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"
Oct 31 19:30:00.490629 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.490613    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73492d9-5852-498a-a6b7-10e2f446feeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84\" (UID: \"f73492d9-5852-498a-a6b7-10e2f446feeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"
Oct 31 19:30:00.490682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.490633    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f886bc5-ea49-4717-8e09-d3fe02c3d334-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp\" (UID: \"0f886bc5-ea49-4717-8e09-d3fe02c3d334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"
Oct 31 19:30:00.497163 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.497126    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"
Oct 31 19:30:00.548380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.548291    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9h9j\" (UniqueName: \"kubernetes.io/projected/f73492d9-5852-498a-a6b7-10e2f446feeb-kube-api-access-f9h9j\") pod \"maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84\" (UID: \"f73492d9-5852-498a-a6b7-10e2f446feeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"
Oct 31 19:30:00.548637 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.548617    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwg2\" (UniqueName: \"kubernetes.io/projected/ba38b8bd-6dcb-4b68-9226-2455456bcf74-kube-api-access-rfwg2\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p\" (UID: \"ba38b8bd-6dcb-4b68-9226-2455456bcf74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"
Oct 31 19:30:00.548988 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.548963    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwkvt\" (UniqueName: \"kubernetes.io/projected/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-kube-api-access-xwkvt\") pod \"maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6\" (UID: \"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"
Oct 31 19:30:00.550204 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.550167    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fms6\" (UniqueName: \"kubernetes.io/projected/0f886bc5-ea49-4717-8e09-d3fe02c3d334-kube-api-access-4fms6\") pod \"maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp\" (UID: \"0f886bc5-ea49-4717-8e09-d3fe02c3d334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"
Oct 31 19:30:00.582585 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.582536    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"
Oct 31 19:30:00.592181 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.592151    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d07c93-21d5-47ad-b7f1-d29c35b31f57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r\" (UID: \"39d07c93-21d5-47ad-b7f1-d29c35b31f57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"
Oct 31 19:30:00.592310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.592208    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6mh\" (UniqueName: \"kubernetes.io/projected/a901ad86-48e1-422b-a342-64e3433fd101-kube-api-access-xk6mh\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb\" (UID: \"a901ad86-48e1-422b-a342-64e3433fd101\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"
Oct 31 19:30:00.592310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.592237    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a901ad86-48e1-422b-a342-64e3433fd101-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb\" (UID: \"a901ad86-48e1-422b-a342-64e3433fd101\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"
Oct 31 19:30:00.592418 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.592331    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0f8eb1-6091-43f3-8928-9b06bad2daae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m\" (UID: \"7c0f8eb1-6091-43f3-8928-9b06bad2daae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"
Oct 31 19:30:00.592537 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.592521    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67kvl\" (UniqueName: \"kubernetes.io/projected/7c0f8eb1-6091-43f3-8928-9b06bad2daae-kube-api-access-67kvl\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m\" (UID: \"7c0f8eb1-6091-43f3-8928-9b06bad2daae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"
Oct 31 19:30:00.592607 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.592574    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8m7p\" (UniqueName: \"kubernetes.io/projected/39d07c93-21d5-47ad-b7f1-d29c35b31f57-kube-api-access-p8m7p\") pod \"maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r\" (UID: \"39d07c93-21d5-47ad-b7f1-d29c35b31f57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"
Oct 31 19:30:00.593392 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.593370    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a901ad86-48e1-422b-a342-64e3433fd101-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb\" (UID: \"a901ad86-48e1-422b-a342-64e3433fd101\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"
Oct 31 19:30:00.610419 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.610394    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"
Oct 31 19:30:00.619714 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.619462    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:00.650590 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.650564    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6mh\" (UniqueName: \"kubernetes.io/projected/a901ad86-48e1-422b-a342-64e3433fd101-kube-api-access-xk6mh\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb\" (UID: \"a901ad86-48e1-422b-a342-64e3433fd101\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"
Oct 31 19:30:00.701796 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.701759    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8m7p\" (UniqueName: \"kubernetes.io/projected/39d07c93-21d5-47ad-b7f1-d29c35b31f57-kube-api-access-p8m7p\") pod \"maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r\" (UID: \"39d07c93-21d5-47ad-b7f1-d29c35b31f57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"
Oct 31 19:30:00.701943 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.701848    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d07c93-21d5-47ad-b7f1-d29c35b31f57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r\" (UID: \"39d07c93-21d5-47ad-b7f1-d29c35b31f57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"
Oct 31 19:30:00.702013 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.701975    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0f8eb1-6091-43f3-8928-9b06bad2daae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m\" (UID: \"7c0f8eb1-6091-43f3-8928-9b06bad2daae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"
Oct 31 19:30:00.702177 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.702156    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67kvl\" (UniqueName: \"kubernetes.io/projected/7c0f8eb1-6091-43f3-8928-9b06bad2daae-kube-api-access-67kvl\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m\" (UID: \"7c0f8eb1-6091-43f3-8928-9b06bad2daae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"
Oct 31 19:30:00.703444 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.703416    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d07c93-21d5-47ad-b7f1-d29c35b31f57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r\" (UID: \"39d07c93-21d5-47ad-b7f1-d29c35b31f57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"
Oct 31 19:30:00.704259 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.704234    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0f8eb1-6091-43f3-8928-9b06bad2daae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m\" (UID: \"7c0f8eb1-6091-43f3-8928-9b06bad2daae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"
Oct 31 19:30:00.710136 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.710117    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"
Oct 31 19:30:00.763961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.763849    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67kvl\" (UniqueName: \"kubernetes.io/projected/7c0f8eb1-6091-43f3-8928-9b06bad2daae-kube-api-access-67kvl\") pod \"maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m\" (UID: \"7c0f8eb1-6091-43f3-8928-9b06bad2daae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"
Oct 31 19:30:00.765019 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.764965    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8m7p\" (UniqueName: \"kubernetes.io/projected/39d07c93-21d5-47ad-b7f1-d29c35b31f57-kube-api-access-p8m7p\") pod \"maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r\" (UID: \"39d07c93-21d5-47ad-b7f1-d29c35b31f57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"
Oct 31 19:30:00.806089 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.806055    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"
Oct 31 19:30:00.814744 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.814704    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"
Oct 31 19:30:00.865509 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.864493    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:00.912202 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.912074    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"]
Oct 31 19:30:00.920515 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:30:00.912608    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5467e676_3261_4bb9_9ef4_b8a296072512.slice/crio-6cc3d2c6a503dc25c7a1de5c6b67003647734e1a93c4b5500b56e881cd882ac9 WatchSource:0}: Error finding container 6cc3d2c6a503dc25c7a1de5c6b67003647734e1a93c4b5500b56e881cd882ac9: Status 404 returned error can't find the container with id 6cc3d2c6a503dc25c7a1de5c6b67003647734e1a93c4b5500b56e881cd882ac9
Oct 31 19:30:00.979177 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:00.979140    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"]
Oct 31 19:30:01.021978 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.021954    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"
Oct 31 19:30:01.022337 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.022317    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"]
Oct 31 19:30:01.030307 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.030287    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"
Oct 31 19:30:01.063549 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:30:01.063523    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba38b8bd_6dcb_4b68_9226_2455456bcf74.slice/crio-a69a4b5c52909452fc78087030f0a5eab90561dcd3635136a1af59047fa7dab9 WatchSource:0}: Error finding container a69a4b5c52909452fc78087030f0a5eab90561dcd3635136a1af59047fa7dab9: Status 404 returned error can't find the container with id a69a4b5c52909452fc78087030f0a5eab90561dcd3635136a1af59047fa7dab9
Oct 31 19:30:01.119618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.119590    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"]
Oct 31 19:30:01.225430 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:30:01.225390    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73492d9_5852_498a_a6b7_10e2f446feeb.slice/crio-8ef29d86227f2a39498ee57293db56dbfaa3baffa323fa2b60c4bb657a49dc4e WatchSource:0}: Error finding container 8ef29d86227f2a39498ee57293db56dbfaa3baffa323fa2b60c4bb657a49dc4e: Status 404 returned error can't find the container with id 8ef29d86227f2a39498ee57293db56dbfaa3baffa323fa2b60c4bb657a49dc4e
Oct 31 19:30:01.225580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.225533    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"]
Oct 31 19:30:01.232655 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.232622    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"]
Oct 31 19:30:01.233831 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:30:01.233792    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bddf03c_e580_4a67_b122_2fb0ebb7b7ac.slice/crio-c0419ca80a0d31690b290352a9a4072b6a46910c82a057dc30d6a5d77b9adfe0 WatchSource:0}: Error finding container c0419ca80a0d31690b290352a9a4072b6a46910c82a057dc30d6a5d77b9adfe0: Status 404 returned error can't find the container with id c0419ca80a0d31690b290352a9a4072b6a46910c82a057dc30d6a5d77b9adfe0
Oct 31 19:30:01.318903 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.318869    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb" event={"ID":"a901ad86-48e1-422b-a342-64e3433fd101","Type":"ContainerStarted","Data":"619c4560a9446748c86550839dc15b93a4c4d5a147da1eb8525c1e132b124359"}
Oct 31 19:30:01.320293 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.320271    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v" event={"ID":"5467e676-3261-4bb9-9ef4-b8a296072512","Type":"ContainerStarted","Data":"6cc3d2c6a503dc25c7a1de5c6b67003647734e1a93c4b5500b56e881cd882ac9"}
Oct 31 19:30:01.321719 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.321693    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp" event={"ID":"0f886bc5-ea49-4717-8e09-d3fe02c3d334","Type":"ContainerStarted","Data":"c2690fcb98b11a5e4841d4d9a9a4322f100251f578710f70bba6a0c54c5096f1"}
Oct 31 19:30:01.323447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.323423    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84" event={"ID":"f73492d9-5852-498a-a6b7-10e2f446feeb","Type":"ContainerStarted","Data":"8ef29d86227f2a39498ee57293db56dbfaa3baffa323fa2b60c4bb657a49dc4e"}
Oct 31 19:30:01.324863 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.324843    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p" event={"ID":"ba38b8bd-6dcb-4b68-9226-2455456bcf74","Type":"ContainerStarted","Data":"a69a4b5c52909452fc78087030f0a5eab90561dcd3635136a1af59047fa7dab9"}
Oct 31 19:30:01.326150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.326133    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6" event={"ID":"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac","Type":"ContainerStarted","Data":"c0419ca80a0d31690b290352a9a4072b6a46910c82a057dc30d6a5d77b9adfe0"}
Oct 31 19:30:01.471734 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.471698    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"]
Oct 31 19:30:01.477170 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:30:01.477137    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0f8eb1_6091_43f3_8928_9b06bad2daae.slice/crio-9df4058c24a8c0c5a2b29bc468e6067461df33fcc9300c976f08cff3783b016a WatchSource:0}: Error finding container 9df4058c24a8c0c5a2b29bc468e6067461df33fcc9300c976f08cff3783b016a: Status 404 returned error can't find the container with id 9df4058c24a8c0c5a2b29bc468e6067461df33fcc9300c976f08cff3783b016a
Oct 31 19:30:01.480813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.480783    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"]
Oct 31 19:30:01.489589 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:30:01.489561    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d07c93_21d5_47ad_b7f1_d29c35b31f57.slice/crio-1f70ee91dd5fe28bfeb02aff344b881acdd418ecaa3e46b908a4e6d654cee1d2 WatchSource:0}: Error finding container 1f70ee91dd5fe28bfeb02aff344b881acdd418ecaa3e46b908a4e6d654cee1d2: Status 404 returned error can't find the container with id 1f70ee91dd5fe28bfeb02aff344b881acdd418ecaa3e46b908a4e6d654cee1d2
Oct 31 19:30:01.834878 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:01.834795    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:02.135440 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.135338    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:02.337705 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.337668    4484 generic.go:356] "Generic (PLEG): container finished" podID="a901ad86-48e1-422b-a342-64e3433fd101" containerID="bb78310a786ce5b6b1f17b2cb5c464c385e432414410cad64e97121b2ade7891" exitCode=60
Oct 31 19:30:02.337913 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.337807    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb" event={"ID":"a901ad86-48e1-422b-a342-64e3433fd101","Type":"ContainerDied","Data":"bb78310a786ce5b6b1f17b2cb5c464c385e432414410cad64e97121b2ade7891"}
Oct 31 19:30:02.338244 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.338221    4484 scope.go:122] "RemoveContainer" containerID="bb78310a786ce5b6b1f17b2cb5c464c385e432414410cad64e97121b2ade7891"
Oct 31 19:30:02.343831 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.343812    4484 generic.go:356] "Generic (PLEG): container finished" podID="5467e676-3261-4bb9-9ef4-b8a296072512" containerID="fd7e2a6e255053c30ed3287668b738521ae2f47f8bd1278cc1aaaeea11c4f231" exitCode=60
Oct 31 19:30:02.343992 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.343836    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v" event={"ID":"5467e676-3261-4bb9-9ef4-b8a296072512","Type":"ContainerDied","Data":"fd7e2a6e255053c30ed3287668b738521ae2f47f8bd1278cc1aaaeea11c4f231"}
Oct 31 19:30:02.344140 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.344124    4484 scope.go:122] "RemoveContainer" containerID="fd7e2a6e255053c30ed3287668b738521ae2f47f8bd1278cc1aaaeea11c4f231"
Oct 31 19:30:02.351455 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.346949    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m" event={"ID":"7c0f8eb1-6091-43f3-8928-9b06bad2daae","Type":"ContainerStarted","Data":"4b3de42b0360ee5e4b98be653cc1bcd807c015394195f7caae03a3d0d52432da"}
Oct 31 19:30:02.351455 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.346999    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m" event={"ID":"7c0f8eb1-6091-43f3-8928-9b06bad2daae","Type":"ContainerStarted","Data":"9df4058c24a8c0c5a2b29bc468e6067461df33fcc9300c976f08cff3783b016a"}
Oct 31 19:30:02.354342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.354315    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r" event={"ID":"39d07c93-21d5-47ad-b7f1-d29c35b31f57","Type":"ContainerStarted","Data":"9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d"}
Oct 31 19:30:02.354457 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.354366    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r" event={"ID":"39d07c93-21d5-47ad-b7f1-d29c35b31f57","Type":"ContainerStarted","Data":"1f70ee91dd5fe28bfeb02aff344b881acdd418ecaa3e46b908a4e6d654cee1d2"}
Oct 31 19:30:02.358321 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.358296    4484 generic.go:356] "Generic (PLEG): container finished" podID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" containerID="68cb0af99728d07827ebdfa0bfb383ab5f17a5b95b605a2b6d97fd3dd2032a27" exitCode=60
Oct 31 19:30:02.358436 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.358391    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp" event={"ID":"0f886bc5-ea49-4717-8e09-d3fe02c3d334","Type":"ContainerDied","Data":"68cb0af99728d07827ebdfa0bfb383ab5f17a5b95b605a2b6d97fd3dd2032a27"}
Oct 31 19:30:02.358650 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.358633    4484 scope.go:122] "RemoveContainer" containerID="68cb0af99728d07827ebdfa0bfb383ab5f17a5b95b605a2b6d97fd3dd2032a27"
Oct 31 19:30:02.371511 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.360467    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84" event={"ID":"f73492d9-5852-498a-a6b7-10e2f446feeb","Type":"ContainerStarted","Data":"223d3461015883ff192dd082a01951a288dd18bcad6c215b63d891ff0a9fe4ef"}
Oct 31 19:30:02.371511 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.363747    4484 generic.go:356] "Generic (PLEG): container finished" podID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" containerID="343b5b0b760853294095a1fc3ba8ac6a38c48ce37ba6b53bd42764c8987a40ba" exitCode=60
Oct 31 19:30:02.371511 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.363884    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p" event={"ID":"ba38b8bd-6dcb-4b68-9226-2455456bcf74","Type":"ContainerDied","Data":"343b5b0b760853294095a1fc3ba8ac6a38c48ce37ba6b53bd42764c8987a40ba"}
Oct 31 19:30:02.371511 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.364187    4484 scope.go:122] "RemoveContainer" containerID="343b5b0b760853294095a1fc3ba8ac6a38c48ce37ba6b53bd42764c8987a40ba"
Oct 31 19:30:02.371511 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.366890    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6" event={"ID":"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac","Type":"ContainerStarted","Data":"5a8e8394e26aa47fd50e85bc1a0ba99edbdb38d1f9572c9e1fd881309165e54b"}
Oct 31 19:30:02.388206 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.388165    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:02.457738 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.457706    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"]
Oct 31 19:30:02.701306 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.701239    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84" podStartSLOduration=2.70121727 podStartE2EDuration="2.70121727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:02.698851694 +0000 UTC m=+4148.845301558" watchObservedRunningTime="2026-10-31 19:30:02.70121727 +0000 UTC m=+4148.847667135"
Oct 31 19:30:02.824527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:02.819775    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6" podStartSLOduration=2.819756304 podStartE2EDuration="2.819756304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:02.816785894 +0000 UTC m=+4148.963235759" watchObservedRunningTime="2026-10-31 19:30:02.819756304 +0000 UTC m=+4148.966206169"
Oct 31 19:30:03.166813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.166764    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m" podStartSLOduration=3.166750233 podStartE2EDuration="3.166750233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:03.165916216 +0000 UTC m=+4149.312366093" watchObservedRunningTime="2026-10-31 19:30:03.166750233 +0000 UTC m=+4149.313200097"
Oct 31 19:30:03.283543 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.283467    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r" podStartSLOduration=3.283454972 podStartE2EDuration="3.283454972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:03.281567933 +0000 UTC m=+4149.428017788" watchObservedRunningTime="2026-10-31 19:30:03.283454972 +0000 UTC m=+4149.429904837"
Oct 31 19:30:03.381915 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.381862    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp" event={"ID":"0f886bc5-ea49-4717-8e09-d3fe02c3d334","Type":"ContainerStarted","Data":"ac5abcedfd05c9415fd7dd7e813d082f5f4907f018313610e00457649b9bb832"}
Oct 31 19:30:03.387444 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.387410    4484 generic.go:356] "Generic (PLEG): container finished" podID="f73492d9-5852-498a-a6b7-10e2f446feeb" containerID="223d3461015883ff192dd082a01951a288dd18bcad6c215b63d891ff0a9fe4ef" exitCode=60
Oct 31 19:30:03.387760 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.387536    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84" event={"ID":"f73492d9-5852-498a-a6b7-10e2f446feeb","Type":"ContainerDied","Data":"223d3461015883ff192dd082a01951a288dd18bcad6c215b63d891ff0a9fe4ef"}
Oct 31 19:30:03.387986 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.387967    4484 scope.go:122] "RemoveContainer" containerID="223d3461015883ff192dd082a01951a288dd18bcad6c215b63d891ff0a9fe4ef"
Oct 31 19:30:03.394954 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.394924    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p" event={"ID":"ba38b8bd-6dcb-4b68-9226-2455456bcf74","Type":"ContainerStarted","Data":"e961d90889c683d00f5ec9b594663c57e32149844652edd4b8c0fdd8b220eedb"}
Oct 31 19:30:03.400695 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.400673    4484 generic.go:356] "Generic (PLEG): container finished" podID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" containerID="5a8e8394e26aa47fd50e85bc1a0ba99edbdb38d1f9572c9e1fd881309165e54b" exitCode=60
Oct 31 19:30:03.400808 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.400785    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6" event={"ID":"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac","Type":"ContainerDied","Data":"5a8e8394e26aa47fd50e85bc1a0ba99edbdb38d1f9572c9e1fd881309165e54b"}
Oct 31 19:30:03.402174 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.401441    4484 scope.go:122] "RemoveContainer" containerID="5a8e8394e26aa47fd50e85bc1a0ba99edbdb38d1f9572c9e1fd881309165e54b"
Oct 31 19:30:03.405113 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.405055    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb" event={"ID":"a901ad86-48e1-422b-a342-64e3433fd101","Type":"ContainerStarted","Data":"ecf9e8465e45c1f5018f94c94450b4803f620867989b06fc411051a86673f80c"}
Oct 31 19:30:03.407210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.407183    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v" event={"ID":"5467e676-3261-4bb9-9ef4-b8a296072512","Type":"ContainerStarted","Data":"c5fae415cc1371d8dab257d311b00904501dbe87c3b1a40e190dbf6e13109ff7"}
Oct 31 19:30:03.408734 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.408708    4484 generic.go:356] "Generic (PLEG): container finished" podID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" containerID="4b3de42b0360ee5e4b98be653cc1bcd807c015394195f7caae03a3d0d52432da" exitCode=60
Oct 31 19:30:03.408841 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.408782    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m" event={"ID":"7c0f8eb1-6091-43f3-8928-9b06bad2daae","Type":"ContainerDied","Data":"4b3de42b0360ee5e4b98be653cc1bcd807c015394195f7caae03a3d0d52432da"}
Oct 31 19:30:03.408970 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.408953    4484 scope.go:122] "RemoveContainer" containerID="4b3de42b0360ee5e4b98be653cc1bcd807c015394195f7caae03a3d0d52432da"
Oct 31 19:30:03.412950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.412010    4484 generic.go:356] "Generic (PLEG): container finished" podID="39d07c93-21d5-47ad-b7f1-d29c35b31f57" containerID="9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d" exitCode=6
Oct 31 19:30:03.412950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.412151    4484 scope.go:122] "RemoveContainer" containerID="9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d"
Oct 31 19:30:03.419905 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.412059    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r" event={"ID":"39d07c93-21d5-47ad-b7f1-d29c35b31f57","Type":"ContainerDied","Data":"9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d"}
Oct 31 19:30:03.498766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.498719    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp" podStartSLOduration=3.498704559 podStartE2EDuration="3.498704559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:03.497606849 +0000 UTC m=+4149.644056713" watchObservedRunningTime="2026-10-31 19:30:03.498704559 +0000 UTC m=+4149.645154422"
Oct 31 19:30:03.530799 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.530765    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:03.846611 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:03.846547    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v" podStartSLOduration=3.8465326600000003 podStartE2EDuration="3.84653266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:03.843979224 +0000 UTC m=+4149.990429084" watchObservedRunningTime="2026-10-31 19:30:03.84653266 +0000 UTC m=+4149.992982523"
Oct 31 19:30:04.191778 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.191686    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb" podStartSLOduration=4.191672416 podStartE2EDuration="4.191672416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:04.189137342 +0000 UTC m=+4150.335587205" watchObservedRunningTime="2026-10-31 19:30:04.191672416 +0000 UTC m=+4150.338122280"
Oct 31 19:30:04.224445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.224413    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:04.306759 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.306710    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p" podStartSLOduration=4.306691485 podStartE2EDuration="4.306691485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:04.305755804 +0000 UTC m=+4150.452205659" watchObservedRunningTime="2026-10-31 19:30:04.306691485 +0000 UTC m=+4150.453141348"
Oct 31 19:30:04.421257 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.421222    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6" event={"ID":"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac","Type":"ContainerStarted","Data":"4c9af3703b494b22c30d381f40a32b6790aa68e78ffe72a61499f519f2ba86cb"}
Oct 31 19:30:04.423264 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.423241    4484 generic.go:356] "Generic (PLEG): container finished" podID="a901ad86-48e1-422b-a342-64e3433fd101" containerID="ecf9e8465e45c1f5018f94c94450b4803f620867989b06fc411051a86673f80c" exitCode=60
Oct 31 19:30:04.423418 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.423331    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb" event={"ID":"a901ad86-48e1-422b-a342-64e3433fd101","Type":"ContainerDied","Data":"ecf9e8465e45c1f5018f94c94450b4803f620867989b06fc411051a86673f80c"}
Oct 31 19:30:04.423418 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.423392    4484 scope.go:122] "RemoveContainer" containerID="bb78310a786ce5b6b1f17b2cb5c464c385e432414410cad64e97121b2ade7891"
Oct 31 19:30:04.423600 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.423582    4484 scope.go:122] "RemoveContainer" containerID="ecf9e8465e45c1f5018f94c94450b4803f620867989b06fc411051a86673f80c"
Oct 31 19:30:04.423917 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:04.423897    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb_odh-ai-gateway-infra(a901ad86-48e1-422b-a342-64e3433fd101)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb" podUID="a901ad86-48e1-422b-a342-64e3433fd101"
Oct 31 19:30:04.425438 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.425416    4484 generic.go:356] "Generic (PLEG): container finished" podID="5467e676-3261-4bb9-9ef4-b8a296072512" containerID="c5fae415cc1371d8dab257d311b00904501dbe87c3b1a40e190dbf6e13109ff7" exitCode=60
Oct 31 19:30:04.425579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.425503    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v" event={"ID":"5467e676-3261-4bb9-9ef4-b8a296072512","Type":"ContainerDied","Data":"c5fae415cc1371d8dab257d311b00904501dbe87c3b1a40e190dbf6e13109ff7"}
Oct 31 19:30:04.425688 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.425675    4484 scope.go:122] "RemoveContainer" containerID="c5fae415cc1371d8dab257d311b00904501dbe87c3b1a40e190dbf6e13109ff7"
Oct 31 19:30:04.425890 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:04.425873    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v_odh-ai-gateway-infra(5467e676-3261-4bb9-9ef4-b8a296072512)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v" podUID="5467e676-3261-4bb9-9ef4-b8a296072512"
Oct 31 19:30:04.427293 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.427267    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m" event={"ID":"7c0f8eb1-6091-43f3-8928-9b06bad2daae","Type":"ContainerStarted","Data":"d77cc407fc6899841bb03fefd28d987c8cd12089d33bff964ff7e25bf7907c69"}
Oct 31 19:30:04.429390 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.429367    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r" event={"ID":"39d07c93-21d5-47ad-b7f1-d29c35b31f57","Type":"ContainerStarted","Data":"22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6"}
Oct 31 19:30:04.429599 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.429387    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r" podUID="39d07c93-21d5-47ad-b7f1-d29c35b31f57" containerName="cleanup" containerID="cri-o://22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6" gracePeriod=30
Oct 31 19:30:04.431429 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.431402    4484 generic.go:356] "Generic (PLEG): container finished" podID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" containerID="ac5abcedfd05c9415fd7dd7e813d082f5f4907f018313610e00457649b9bb832" exitCode=60
Oct 31 19:30:04.431565 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.431544    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp" event={"ID":"0f886bc5-ea49-4717-8e09-d3fe02c3d334","Type":"ContainerDied","Data":"ac5abcedfd05c9415fd7dd7e813d082f5f4907f018313610e00457649b9bb832"}
Oct 31 19:30:04.431836 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.431814    4484 scope.go:122] "RemoveContainer" containerID="ac5abcedfd05c9415fd7dd7e813d082f5f4907f018313610e00457649b9bb832"
Oct 31 19:30:04.432082 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:04.432060    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp_odh-ai-gateway-infra(0f886bc5-ea49-4717-8e09-d3fe02c3d334)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334"
Oct 31 19:30:04.433565 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.433546    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84" event={"ID":"f73492d9-5852-498a-a6b7-10e2f446feeb","Type":"ContainerStarted","Data":"4bc79661e2178f3a9c7d41c2f2b6acb206a64b831b45cfc4e378697855dfbfe0"}
Oct 31 19:30:04.435488 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.435455    4484 generic.go:356] "Generic (PLEG): container finished" podID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" containerID="e961d90889c683d00f5ec9b594663c57e32149844652edd4b8c0fdd8b220eedb" exitCode=60
Oct 31 19:30:04.435576 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.435541    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p" event={"ID":"ba38b8bd-6dcb-4b68-9226-2455456bcf74","Type":"ContainerDied","Data":"e961d90889c683d00f5ec9b594663c57e32149844652edd4b8c0fdd8b220eedb"}
Oct 31 19:30:04.435703 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.435689    4484 scope.go:122] "RemoveContainer" containerID="e961d90889c683d00f5ec9b594663c57e32149844652edd4b8c0fdd8b220eedb"
Oct 31 19:30:04.435877 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:04.435860    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p_odh-ai-gateway-infra(ba38b8bd-6dcb-4b68-9226-2455456bcf74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74"
Oct 31 19:30:04.437736 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.437718    4484 scope.go:122] "RemoveContainer" containerID="fd7e2a6e255053c30ed3287668b738521ae2f47f8bd1278cc1aaaeea11c4f231"
Oct 31 19:30:04.450665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.450650    4484 scope.go:122] "RemoveContainer" containerID="68cb0af99728d07827ebdfa0bfb383ab5f17a5b95b605a2b6d97fd3dd2032a27"
Oct 31 19:30:04.464370 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.464343    4484 scope.go:122] "RemoveContainer" containerID="343b5b0b760853294095a1fc3ba8ac6a38c48ce37ba6b53bd42764c8987a40ba"
Oct 31 19:30:04.762003 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.761972    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"
Oct 31 19:30:04.864494 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.864439    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d07c93-21d5-47ad-b7f1-d29c35b31f57-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d07c93-21d5-47ad-b7f1-d29c35b31f57-maas-api-service-ca\") pod \"39d07c93-21d5-47ad-b7f1-d29c35b31f57\" (UID: \"39d07c93-21d5-47ad-b7f1-d29c35b31f57\") "
Oct 31 19:30:04.864927 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.864696    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d07c93-21d5-47ad-b7f1-d29c35b31f57-kube-api-access-p8m7p\" (UniqueName: \"kubernetes.io/projected/39d07c93-21d5-47ad-b7f1-d29c35b31f57-kube-api-access-p8m7p\") pod \"39d07c93-21d5-47ad-b7f1-d29c35b31f57\" (UID: \"39d07c93-21d5-47ad-b7f1-d29c35b31f57\") "
Oct 31 19:30:04.864996 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.864922    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d07c93-21d5-47ad-b7f1-d29c35b31f57-maas-api-service-ca" pod "39d07c93-21d5-47ad-b7f1-d29c35b31f57" (UID: "39d07c93-21d5-47ad-b7f1-d29c35b31f57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:04.865760 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.865293    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39d07c93-21d5-47ad-b7f1-d29c35b31f57-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:04.867245 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.867224    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d07c93-21d5-47ad-b7f1-d29c35b31f57-kube-api-access-p8m7p" pod "39d07c93-21d5-47ad-b7f1-d29c35b31f57" (UID: "39d07c93-21d5-47ad-b7f1-d29c35b31f57"). InnerVolumeSpecName "kube-api-access-p8m7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:04.967151 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:04.967111    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8m7p\" (UniqueName: \"kubernetes.io/projected/39d07c93-21d5-47ad-b7f1-d29c35b31f57-kube-api-access-p8m7p\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:05.449021 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.448982    4484 generic.go:356] "Generic (PLEG): container finished" podID="f73492d9-5852-498a-a6b7-10e2f446feeb" containerID="4bc79661e2178f3a9c7d41c2f2b6acb206a64b831b45cfc4e378697855dfbfe0" exitCode=60
Oct 31 19:30:05.449213 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.449063    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84" event={"ID":"f73492d9-5852-498a-a6b7-10e2f446feeb","Type":"ContainerDied","Data":"4bc79661e2178f3a9c7d41c2f2b6acb206a64b831b45cfc4e378697855dfbfe0"}
Oct 31 19:30:05.449213 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.449117    4484 scope.go:122] "RemoveContainer" containerID="223d3461015883ff192dd082a01951a288dd18bcad6c215b63d891ff0a9fe4ef"
Oct 31 19:30:05.449442 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.449422    4484 scope.go:122] "RemoveContainer" containerID="4bc79661e2178f3a9c7d41c2f2b6acb206a64b831b45cfc4e378697855dfbfe0"
Oct 31 19:30:05.449822 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:05.449792    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84_odh-ai-gateway-infra(f73492d9-5852-498a-a6b7-10e2f446feeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb"
Oct 31 19:30:05.451298 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.451280    4484 scope.go:122] "RemoveContainer" containerID="e961d90889c683d00f5ec9b594663c57e32149844652edd4b8c0fdd8b220eedb"
Oct 31 19:30:05.451598 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:05.451576    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p_odh-ai-gateway-infra(ba38b8bd-6dcb-4b68-9226-2455456bcf74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74"
Oct 31 19:30:05.452834 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.452814    4484 generic.go:356] "Generic (PLEG): container finished" podID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" containerID="4c9af3703b494b22c30d381f40a32b6790aa68e78ffe72a61499f519f2ba86cb" exitCode=60
Oct 31 19:30:05.452950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.452844    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6" event={"ID":"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac","Type":"ContainerDied","Data":"4c9af3703b494b22c30d381f40a32b6790aa68e78ffe72a61499f519f2ba86cb"}
Oct 31 19:30:05.453133 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.453112    4484 scope.go:122] "RemoveContainer" containerID="4c9af3703b494b22c30d381f40a32b6790aa68e78ffe72a61499f519f2ba86cb"
Oct 31 19:30:05.453444 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:05.453422    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6_odh-ai-gateway-infra(5bddf03c-e580-4a67-b122-2fb0ebb7b7ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac"
Oct 31 19:30:05.454827 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.454812    4484 scope.go:122] "RemoveContainer" containerID="ecf9e8465e45c1f5018f94c94450b4803f620867989b06fc411051a86673f80c"
Oct 31 19:30:05.455074 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:05.455050    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb_odh-ai-gateway-infra(a901ad86-48e1-422b-a342-64e3433fd101)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb" podUID="a901ad86-48e1-422b-a342-64e3433fd101"
Oct 31 19:30:05.456577 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.456560    4484 scope.go:122] "RemoveContainer" containerID="c5fae415cc1371d8dab257d311b00904501dbe87c3b1a40e190dbf6e13109ff7"
Oct 31 19:30:05.456808 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:05.456784    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v_odh-ai-gateway-infra(5467e676-3261-4bb9-9ef4-b8a296072512)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v" podUID="5467e676-3261-4bb9-9ef4-b8a296072512"
Oct 31 19:30:05.458230 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.458207    4484 generic.go:356] "Generic (PLEG): container finished" podID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" containerID="d77cc407fc6899841bb03fefd28d987c8cd12089d33bff964ff7e25bf7907c69" exitCode=60
Oct 31 19:30:05.458321 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.458290    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m" event={"ID":"7c0f8eb1-6091-43f3-8928-9b06bad2daae","Type":"ContainerDied","Data":"d77cc407fc6899841bb03fefd28d987c8cd12089d33bff964ff7e25bf7907c69"}
Oct 31 19:30:05.458467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.458453    4484 scope.go:122] "RemoveContainer" containerID="d77cc407fc6899841bb03fefd28d987c8cd12089d33bff964ff7e25bf7907c69"
Oct 31 19:30:05.458717 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:05.458697    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m_odh-ai-gateway-infra(7c0f8eb1-6091-43f3-8928-9b06bad2daae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae"
Oct 31 19:30:05.459864 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.459842    4484 generic.go:356] "Generic (PLEG): container finished" podID="39d07c93-21d5-47ad-b7f1-d29c35b31f57" containerID="22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6" exitCode=6
Oct 31 19:30:05.459979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.459872    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r" event={"ID":"39d07c93-21d5-47ad-b7f1-d29c35b31f57","Type":"ContainerDied","Data":"22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6"}
Oct 31 19:30:05.459979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.459895    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"
Oct 31 19:30:05.459979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.459926    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r" event={"ID":"39d07c93-21d5-47ad-b7f1-d29c35b31f57","Type":"ContainerDied","Data":"1f70ee91dd5fe28bfeb02aff344b881acdd418ecaa3e46b908a4e6d654cee1d2"}
Oct 31 19:30:05.461870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.461854    4484 scope.go:122] "RemoveContainer" containerID="ac5abcedfd05c9415fd7dd7e813d082f5f4907f018313610e00457649b9bb832"
Oct 31 19:30:05.462077 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:05.462056    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp_odh-ai-gateway-infra(0f886bc5-ea49-4717-8e09-d3fe02c3d334)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334"
Oct 31 19:30:05.463816 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.463800    4484 scope.go:122] "RemoveContainer" containerID="5a8e8394e26aa47fd50e85bc1a0ba99edbdb38d1f9572c9e1fd881309165e54b"
Oct 31 19:30:05.485924 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.485905    4484 scope.go:122] "RemoveContainer" containerID="4b3de42b0360ee5e4b98be653cc1bcd807c015394195f7caae03a3d0d52432da"
Oct 31 19:30:05.504552 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.504532    4484 scope.go:122] "RemoveContainer" containerID="22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6"
Oct 31 19:30:05.515451 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.515424    4484 scope.go:122] "RemoveContainer" containerID="9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d"
Oct 31 19:30:05.527060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.527039    4484 scope.go:122] "RemoveContainer" containerID="22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6"
Oct 31 19:30:05.527342 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:05.527313    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6\": container with ID starting with 22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6 not found: ID does not exist" containerID="22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6"
Oct 31 19:30:05.527442 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.527353    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6"} err="failed to get container status \"22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6\": rpc error: code = NotFound desc = could not find container \"22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6\": container with ID starting with 22f2680daccd53eb4228070f2ec619e4129352f1d2cfa737754ca9156efbbca6 not found: ID does not exist"
Oct 31 19:30:05.527442 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.527380    4484 scope.go:122] "RemoveContainer" containerID="9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d"
Oct 31 19:30:05.527679 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:05.527656    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d\": container with ID starting with 9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d not found: ID does not exist" containerID="9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d"
Oct 31 19:30:05.527747 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:05.527689    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d"} err="failed to get container status \"9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d\": rpc error: code = NotFound desc = could not find container \"9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d\": container with ID starting with 9b683c32d92d8e6741f48c0bebdc6203a238b2e7df096362cf0db9183288208d not found: ID does not exist"
Oct 31 19:30:06.065796 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:06.065760    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"]
Oct 31 19:30:06.069867 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:06.069834    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-227f0f-29891250-s6l4r"]
Oct 31 19:30:06.472589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:06.472499    4484 scope.go:122] "RemoveContainer" containerID="4bc79661e2178f3a9c7d41c2f2b6acb206a64b831b45cfc4e378697855dfbfe0"
Oct 31 19:30:06.472860 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:06.472828    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84_odh-ai-gateway-infra(f73492d9-5852-498a-a6b7-10e2f446feeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb"
Oct 31 19:30:06.474201 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:06.474181    4484 scope.go:122] "RemoveContainer" containerID="4c9af3703b494b22c30d381f40a32b6790aa68e78ffe72a61499f519f2ba86cb"
Oct 31 19:30:06.474498 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:06.474453    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6_odh-ai-gateway-infra(5bddf03c-e580-4a67-b122-2fb0ebb7b7ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac"
Oct 31 19:30:06.475625 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:06.475609    4484 scope.go:122] "RemoveContainer" containerID="d77cc407fc6899841bb03fefd28d987c8cd12089d33bff964ff7e25bf7907c69"
Oct 31 19:30:06.475883 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:06.475858    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m_odh-ai-gateway-infra(7c0f8eb1-6091-43f3-8928-9b06bad2daae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae"
Oct 31 19:30:07.092803 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:07.092772    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:07.334792 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:07.334755    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:07.995234 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:07.995204    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d07c93-21d5-47ad-b7f1-d29c35b31f57" path="/var/lib/kubelet/pods/39d07c93-21d5-47ad-b7f1-d29c35b31f57/volumes"
Oct 31 19:30:08.029810 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:08.029781    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:08.659128 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:08.659094    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:09.173058 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:09.173028    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:09.443752 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:09.443661    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:11.848628 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:11.848595    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:12.167574 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:12.167463    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:13.174414 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:13.174369    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:13.547506 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:13.547446    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:13.991570 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:13.991533    4484 scope.go:122] "RemoveContainer" containerID="e961d90889c683d00f5ec9b594663c57e32149844652edd4b8c0fdd8b220eedb"
Oct 31 19:30:14.248663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:14.248576    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:14.532347 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:14.532264    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p" event={"ID":"ba38b8bd-6dcb-4b68-9226-2455456bcf74","Type":"ContainerStarted","Data":"8117f0f7918f0937496129d83ba94d6cdf4b64d8dd8af78495dabd9a1d0bf7f8"}
Oct 31 19:30:15.541156 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:15.541127    4484 generic.go:356] "Generic (PLEG): container finished" podID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" containerID="8117f0f7918f0937496129d83ba94d6cdf4b64d8dd8af78495dabd9a1d0bf7f8" exitCode=60
Oct 31 19:30:15.541660 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:15.541200    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p" event={"ID":"ba38b8bd-6dcb-4b68-9226-2455456bcf74","Type":"ContainerDied","Data":"8117f0f7918f0937496129d83ba94d6cdf4b64d8dd8af78495dabd9a1d0bf7f8"}
Oct 31 19:30:15.541660 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:15.541265    4484 scope.go:122] "RemoveContainer" containerID="e961d90889c683d00f5ec9b594663c57e32149844652edd4b8c0fdd8b220eedb"
Oct 31 19:30:15.541769 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:15.541682    4484 scope.go:122] "RemoveContainer" containerID="8117f0f7918f0937496129d83ba94d6cdf4b64d8dd8af78495dabd9a1d0bf7f8"
Oct 31 19:30:15.542024 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:15.542004    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p_odh-ai-gateway-infra(ba38b8bd-6dcb-4b68-9226-2455456bcf74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74"
Oct 31 19:30:15.664137 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:15.664107    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"]
Oct 31 19:30:15.986670 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:15.986642    4484 scope.go:122] "RemoveContainer" containerID="ac5abcedfd05c9415fd7dd7e813d082f5f4907f018313610e00457649b9bb832"
Oct 31 19:30:16.550434 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:16.550390    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp" event={"ID":"0f886bc5-ea49-4717-8e09-d3fe02c3d334","Type":"ContainerStarted","Data":"2f016d803c436cb886a5a7d71e2f8d8c262ed5662274cc0339e39b0efc7b03e8"}
Oct 31 19:30:16.702730 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:16.702699    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"
Oct 31 19:30:16.812267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:16.812239    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba38b8bd-6dcb-4b68-9226-2455456bcf74-kube-api-access-rfwg2\" (UniqueName: \"kubernetes.io/projected/ba38b8bd-6dcb-4b68-9226-2455456bcf74-kube-api-access-rfwg2\") pod \"ba38b8bd-6dcb-4b68-9226-2455456bcf74\" (UID: \"ba38b8bd-6dcb-4b68-9226-2455456bcf74\") "
Oct 31 19:30:16.812464 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:16.812280    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba38b8bd-6dcb-4b68-9226-2455456bcf74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba38b8bd-6dcb-4b68-9226-2455456bcf74-maas-api-service-ca\") pod \"ba38b8bd-6dcb-4b68-9226-2455456bcf74\" (UID: \"ba38b8bd-6dcb-4b68-9226-2455456bcf74\") "
Oct 31 19:30:16.812735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:16.812705    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba38b8bd-6dcb-4b68-9226-2455456bcf74-maas-api-service-ca" pod "ba38b8bd-6dcb-4b68-9226-2455456bcf74" (UID: "ba38b8bd-6dcb-4b68-9226-2455456bcf74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:16.814571 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:16.814544    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba38b8bd-6dcb-4b68-9226-2455456bcf74-kube-api-access-rfwg2" pod "ba38b8bd-6dcb-4b68-9226-2455456bcf74" (UID: "ba38b8bd-6dcb-4b68-9226-2455456bcf74"). InnerVolumeSpecName "kube-api-access-rfwg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:16.914196 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:16.914166    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfwg2\" (UniqueName: \"kubernetes.io/projected/ba38b8bd-6dcb-4b68-9226-2455456bcf74-kube-api-access-rfwg2\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:16.914196 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:16.914196    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba38b8bd-6dcb-4b68-9226-2455456bcf74-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:16.987540 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:16.987512    4484 scope.go:122] "RemoveContainer" containerID="ecf9e8465e45c1f5018f94c94450b4803f620867989b06fc411051a86673f80c"
Oct 31 19:30:16.987646 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:16.987558    4484 scope.go:122] "RemoveContainer" containerID="c5fae415cc1371d8dab257d311b00904501dbe87c3b1a40e190dbf6e13109ff7"
Oct 31 19:30:17.564008 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.563913    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb" event={"ID":"a901ad86-48e1-422b-a342-64e3433fd101","Type":"ContainerStarted","Data":"a69b285ffc581eab11287509cb481558ca9f9aa07fa6f966c80b94bc7b7bcf23"}
Oct 31 19:30:17.566122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.566096    4484 generic.go:356] "Generic (PLEG): container finished" podID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" containerID="2f016d803c436cb886a5a7d71e2f8d8c262ed5662274cc0339e39b0efc7b03e8" exitCode=60
Oct 31 19:30:17.566273 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.566164    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp" event={"ID":"0f886bc5-ea49-4717-8e09-d3fe02c3d334","Type":"ContainerDied","Data":"2f016d803c436cb886a5a7d71e2f8d8c262ed5662274cc0339e39b0efc7b03e8"}
Oct 31 19:30:17.566273 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.566222    4484 scope.go:122] "RemoveContainer" containerID="ac5abcedfd05c9415fd7dd7e813d082f5f4907f018313610e00457649b9bb832"
Oct 31 19:30:17.566613 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.566596    4484 scope.go:122] "RemoveContainer" containerID="2f016d803c436cb886a5a7d71e2f8d8c262ed5662274cc0339e39b0efc7b03e8"
Oct 31 19:30:17.566936 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:17.566913    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp_odh-ai-gateway-infra(0f886bc5-ea49-4717-8e09-d3fe02c3d334)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334"
Oct 31 19:30:17.567813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.567791    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p" event={"ID":"ba38b8bd-6dcb-4b68-9226-2455456bcf74","Type":"ContainerDied","Data":"a69a4b5c52909452fc78087030f0a5eab90561dcd3635136a1af59047fa7dab9"}
Oct 31 19:30:17.567904 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.567824    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"
Oct 31 19:30:17.580051 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.580035    4484 scope.go:122] "RemoveContainer" containerID="8117f0f7918f0937496129d83ba94d6cdf4b64d8dd8af78495dabd9a1d0bf7f8"
Oct 31 19:30:17.686027 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.685989    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"]
Oct 31 19:30:17.800783 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.800753    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"]
Oct 31 19:30:17.809920 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.809887    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4731a-29891250-n6j9p"]
Oct 31 19:30:17.986328 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.986306    4484 scope.go:122] "RemoveContainer" containerID="4bc79661e2178f3a9c7d41c2f2b6acb206a64b831b45cfc4e378697855dfbfe0"
Oct 31 19:30:17.986524 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.986414    4484 scope.go:122] "RemoveContainer" containerID="4c9af3703b494b22c30d381f40a32b6790aa68e78ffe72a61499f519f2ba86cb"
Oct 31 19:30:17.999703 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:17.999681    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" path="/var/lib/kubelet/pods/ba38b8bd-6dcb-4b68-9226-2455456bcf74/volumes"
Oct 31 19:30:18.378883 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.378852    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:18.576919 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.576838    4484 generic.go:356] "Generic (PLEG): container finished" podID="5467e676-3261-4bb9-9ef4-b8a296072512" containerID="c8a73fe2e481160d1e88f1acc6dc1485542ff710a00e123a42a32163c59396b3" exitCode=60
Oct 31 19:30:18.576919 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.576908    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v" event={"ID":"5467e676-3261-4bb9-9ef4-b8a296072512","Type":"ContainerDied","Data":"c8a73fe2e481160d1e88f1acc6dc1485542ff710a00e123a42a32163c59396b3"}
Oct 31 19:30:18.577447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.576963    4484 scope.go:122] "RemoveContainer" containerID="c5fae415cc1371d8dab257d311b00904501dbe87c3b1a40e190dbf6e13109ff7"
Oct 31 19:30:18.577447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.577360    4484 scope.go:122] "RemoveContainer" containerID="c8a73fe2e481160d1e88f1acc6dc1485542ff710a00e123a42a32163c59396b3"
Oct 31 19:30:18.577735 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:18.577712    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v_odh-ai-gateway-infra(5467e676-3261-4bb9-9ef4-b8a296072512)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v" podUID="5467e676-3261-4bb9-9ef4-b8a296072512"
Oct 31 19:30:18.580990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.580968    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84" event={"ID":"f73492d9-5852-498a-a6b7-10e2f446feeb","Type":"ContainerStarted","Data":"0f334596f21b21dfb29c6276165e1278daa6a9e30030da70b10abf434cf547ad"}
Oct 31 19:30:18.584335 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.584312    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6" event={"ID":"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac","Type":"ContainerStarted","Data":"e35b9846cf32dbf72ef695ad60218114917367954a83f648611a0359b7fd8aca"}
Oct 31 19:30:18.586391 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.586368    4484 generic.go:356] "Generic (PLEG): container finished" podID="a901ad86-48e1-422b-a342-64e3433fd101" containerID="a69b285ffc581eab11287509cb481558ca9f9aa07fa6f966c80b94bc7b7bcf23" exitCode=60
Oct 31 19:30:18.586519 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.586410    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb" event={"ID":"a901ad86-48e1-422b-a342-64e3433fd101","Type":"ContainerDied","Data":"a69b285ffc581eab11287509cb481558ca9f9aa07fa6f966c80b94bc7b7bcf23"}
Oct 31 19:30:18.586832 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.586815    4484 scope.go:122] "RemoveContainer" containerID="a69b285ffc581eab11287509cb481558ca9f9aa07fa6f966c80b94bc7b7bcf23"
Oct 31 19:30:18.587172 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:18.587149    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb_odh-ai-gateway-infra(a901ad86-48e1-422b-a342-64e3433fd101)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb" podUID="a901ad86-48e1-422b-a342-64e3433fd101"
Oct 31 19:30:18.590859 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.590837    4484 scope.go:122] "RemoveContainer" containerID="ecf9e8465e45c1f5018f94c94450b4803f620867989b06fc411051a86673f80c"
Oct 31 19:30:18.632062 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.632010    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:18.702620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.702351    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"]
Oct 31 19:30:18.759127 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.759097    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"
Oct 31 19:30:18.847568 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.847488    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f886bc5-ea49-4717-8e09-d3fe02c3d334-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f886bc5-ea49-4717-8e09-d3fe02c3d334-maas-api-service-ca\") pod \"0f886bc5-ea49-4717-8e09-d3fe02c3d334\" (UID: \"0f886bc5-ea49-4717-8e09-d3fe02c3d334\") "
Oct 31 19:30:18.847705 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.847638    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f886bc5-ea49-4717-8e09-d3fe02c3d334-kube-api-access-4fms6\" (UniqueName: \"kubernetes.io/projected/0f886bc5-ea49-4717-8e09-d3fe02c3d334-kube-api-access-4fms6\") pod \"0f886bc5-ea49-4717-8e09-d3fe02c3d334\" (UID: \"0f886bc5-ea49-4717-8e09-d3fe02c3d334\") "
Oct 31 19:30:18.847900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.847880    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f886bc5-ea49-4717-8e09-d3fe02c3d334-maas-api-service-ca" pod "0f886bc5-ea49-4717-8e09-d3fe02c3d334" (UID: "0f886bc5-ea49-4717-8e09-d3fe02c3d334"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:18.848156 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.848134    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f886bc5-ea49-4717-8e09-d3fe02c3d334-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:18.850208 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.850183    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f886bc5-ea49-4717-8e09-d3fe02c3d334-kube-api-access-4fms6" pod "0f886bc5-ea49-4717-8e09-d3fe02c3d334" (UID: "0f886bc5-ea49-4717-8e09-d3fe02c3d334"). InnerVolumeSpecName "kube-api-access-4fms6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:18.949753 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.949715    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fms6\" (UniqueName: \"kubernetes.io/projected/0f886bc5-ea49-4717-8e09-d3fe02c3d334-kube-api-access-4fms6\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:18.986768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:18.986742    4484 scope.go:122] "RemoveContainer" containerID="d77cc407fc6899841bb03fefd28d987c8cd12089d33bff964ff7e25bf7907c69"
Oct 31 19:30:19.596188 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.596095    4484 generic.go:356] "Generic (PLEG): container finished" podID="f73492d9-5852-498a-a6b7-10e2f446feeb" containerID="0f334596f21b21dfb29c6276165e1278daa6a9e30030da70b10abf434cf547ad" exitCode=60
Oct 31 19:30:19.596690 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.596183    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84" event={"ID":"f73492d9-5852-498a-a6b7-10e2f446feeb","Type":"ContainerDied","Data":"0f334596f21b21dfb29c6276165e1278daa6a9e30030da70b10abf434cf547ad"}
Oct 31 19:30:19.596690 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.596294    4484 scope.go:122] "RemoveContainer" containerID="4bc79661e2178f3a9c7d41c2f2b6acb206a64b831b45cfc4e378697855dfbfe0"
Oct 31 19:30:19.596805 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.596787    4484 scope.go:122] "RemoveContainer" containerID="0f334596f21b21dfb29c6276165e1278daa6a9e30030da70b10abf434cf547ad"
Oct 31 19:30:19.597176 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:19.597152    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84_odh-ai-gateway-infra(f73492d9-5852-498a-a6b7-10e2f446feeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb"
Oct 31 19:30:19.599128 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.598719    4484 generic.go:356] "Generic (PLEG): container finished" podID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" containerID="e35b9846cf32dbf72ef695ad60218114917367954a83f648611a0359b7fd8aca" exitCode=60
Oct 31 19:30:19.599128 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.598801    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6" event={"ID":"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac","Type":"ContainerDied","Data":"e35b9846cf32dbf72ef695ad60218114917367954a83f648611a0359b7fd8aca"}
Oct 31 19:30:19.599258 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.599171    4484 scope.go:122] "RemoveContainer" containerID="e35b9846cf32dbf72ef695ad60218114917367954a83f648611a0359b7fd8aca"
Oct 31 19:30:19.599504 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:19.599461    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6_odh-ai-gateway-infra(5bddf03c-e580-4a67-b122-2fb0ebb7b7ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac"
Oct 31 19:30:19.605770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.605744    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m" event={"ID":"7c0f8eb1-6091-43f3-8928-9b06bad2daae","Type":"ContainerStarted","Data":"702e2ed7a7f171f4ea296e6d52abe6933cc6095303bb5c65edf42c993f76d613"}
Oct 31 19:30:19.607913 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.607889    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp" event={"ID":"0f886bc5-ea49-4717-8e09-d3fe02c3d334","Type":"ContainerDied","Data":"c2690fcb98b11a5e4841d4d9a9a4322f100251f578710f70bba6a0c54c5096f1"}
Oct 31 19:30:19.608022 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.607943    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"
Oct 31 19:30:19.611817 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.611799    4484 scope.go:122] "RemoveContainer" containerID="4c9af3703b494b22c30d381f40a32b6790aa68e78ffe72a61499f519f2ba86cb"
Oct 31 19:30:19.626949 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.626928    4484 scope.go:122] "RemoveContainer" containerID="2f016d803c436cb886a5a7d71e2f8d8c262ed5662274cc0339e39b0efc7b03e8"
Oct 31 19:30:19.708144 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.708110    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"]
Oct 31 19:30:19.768630 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.768602    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"
Oct 31 19:30:19.859429 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.859399    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a901ad86-48e1-422b-a342-64e3433fd101-kube-api-access-xk6mh\" (UniqueName: \"kubernetes.io/projected/a901ad86-48e1-422b-a342-64e3433fd101-kube-api-access-xk6mh\") pod \"a901ad86-48e1-422b-a342-64e3433fd101\" (UID: \"a901ad86-48e1-422b-a342-64e3433fd101\") "
Oct 31 19:30:19.859614 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.859582    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a901ad86-48e1-422b-a342-64e3433fd101-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a901ad86-48e1-422b-a342-64e3433fd101-maas-api-service-ca\") pod \"a901ad86-48e1-422b-a342-64e3433fd101\" (UID: \"a901ad86-48e1-422b-a342-64e3433fd101\") "
Oct 31 19:30:19.860017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.859951    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a901ad86-48e1-422b-a342-64e3433fd101-maas-api-service-ca" pod "a901ad86-48e1-422b-a342-64e3433fd101" (UID: "a901ad86-48e1-422b-a342-64e3433fd101"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:19.860081 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.860064    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a901ad86-48e1-422b-a342-64e3433fd101-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:19.861946 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.861924    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a901ad86-48e1-422b-a342-64e3433fd101-kube-api-access-xk6mh" pod "a901ad86-48e1-422b-a342-64e3433fd101" (UID: "a901ad86-48e1-422b-a342-64e3433fd101"). InnerVolumeSpecName "kube-api-access-xk6mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:19.864904 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.864880    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"
Oct 31 19:30:19.937642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.937611    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"]
Oct 31 19:30:19.960603 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.960571    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5467e676-3261-4bb9-9ef4-b8a296072512-kube-api-access-6lxk5\" (UniqueName: \"kubernetes.io/projected/5467e676-3261-4bb9-9ef4-b8a296072512-kube-api-access-6lxk5\") pod \"5467e676-3261-4bb9-9ef4-b8a296072512\" (UID: \"5467e676-3261-4bb9-9ef4-b8a296072512\") "
Oct 31 19:30:19.960805 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.960788    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5467e676-3261-4bb9-9ef4-b8a296072512-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5467e676-3261-4bb9-9ef4-b8a296072512-maas-api-service-ca\") pod \"5467e676-3261-4bb9-9ef4-b8a296072512\" (UID: \"5467e676-3261-4bb9-9ef4-b8a296072512\") "
Oct 31 19:30:19.961247 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.961226    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk6mh\" (UniqueName: \"kubernetes.io/projected/a901ad86-48e1-422b-a342-64e3433fd101-kube-api-access-xk6mh\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:19.961369 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.961349    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5467e676-3261-4bb9-9ef4-b8a296072512-maas-api-service-ca" pod "5467e676-3261-4bb9-9ef4-b8a296072512" (UID: "5467e676-3261-4bb9-9ef4-b8a296072512"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:19.963315 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:19.963272    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5467e676-3261-4bb9-9ef4-b8a296072512-kube-api-access-6lxk5" pod "5467e676-3261-4bb9-9ef4-b8a296072512" (UID: "5467e676-3261-4bb9-9ef4-b8a296072512"). InnerVolumeSpecName "kube-api-access-6lxk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:20.054294 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.054269    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"]
Oct 31 19:30:20.062268 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.062238    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lxk5\" (UniqueName: \"kubernetes.io/projected/5467e676-3261-4bb9-9ef4-b8a296072512-kube-api-access-6lxk5\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:20.062379 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.062274    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5467e676-3261-4bb9-9ef4-b8a296072512-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:20.169974 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.169896    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:20.316594 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.316564    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"]
Oct 31 19:30:20.320089 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.320059    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c249cd-29891250-8d6pp"]
Oct 31 19:30:20.619059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.619025    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"
Oct 31 19:30:20.619431 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.619052    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb" event={"ID":"a901ad86-48e1-422b-a342-64e3433fd101","Type":"ContainerDied","Data":"619c4560a9446748c86550839dc15b93a4c4d5a147da1eb8525c1e132b124359"}
Oct 31 19:30:20.619431 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.619118    4484 scope.go:122] "RemoveContainer" containerID="a69b285ffc581eab11287509cb481558ca9f9aa07fa6f966c80b94bc7b7bcf23"
Oct 31 19:30:20.620929 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.620891    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"
Oct 31 19:30:20.621117 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.621099    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:20.622020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.621988    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v" event={"ID":"5467e676-3261-4bb9-9ef4-b8a296072512","Type":"ContainerDied","Data":"6cc3d2c6a503dc25c7a1de5c6b67003647734e1a93c4b5500b56e881cd882ac9"}
Oct 31 19:30:20.623670 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.623645    4484 generic.go:356] "Generic (PLEG): container finished" podID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" containerID="702e2ed7a7f171f4ea296e6d52abe6933cc6095303bb5c65edf42c993f76d613" exitCode=60
Oct 31 19:30:20.623820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.623778    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m" event={"ID":"7c0f8eb1-6091-43f3-8928-9b06bad2daae","Type":"ContainerDied","Data":"702e2ed7a7f171f4ea296e6d52abe6933cc6095303bb5c65edf42c993f76d613"}
Oct 31 19:30:20.624123 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.624105    4484 scope.go:122] "RemoveContainer" containerID="702e2ed7a7f171f4ea296e6d52abe6933cc6095303bb5c65edf42c993f76d613"
Oct 31 19:30:20.624403 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:20.624377    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m_odh-ai-gateway-infra(7c0f8eb1-6091-43f3-8928-9b06bad2daae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae"
Oct 31 19:30:20.632130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.632094    4484 scope.go:122] "RemoveContainer" containerID="c8a73fe2e481160d1e88f1acc6dc1485542ff710a00e123a42a32163c59396b3"
Oct 31 19:30:20.643030 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.643004    4484 scope.go:122] "RemoveContainer" containerID="d77cc407fc6899841bb03fefd28d987c8cd12089d33bff964ff7e25bf7907c69"
Oct 31 19:30:20.796706 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.796683    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"
Oct 31 19:30:20.841232 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.841211    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"
Oct 31 19:30:20.851228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.851207    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"]
Oct 31 19:30:20.855922 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.855899    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"]
Oct 31 19:30:20.867279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.867253    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e4a2c-29891250-h9qrb"]
Oct 31 19:30:20.887603 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.887548    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:20.977525 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.977459    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-maas-api-service-ca\") pod \"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac\" (UID: \"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac\") "
Oct 31 19:30:20.977728 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.977642    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73492d9-5852-498a-a6b7-10e2f446feeb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73492d9-5852-498a-a6b7-10e2f446feeb-maas-api-service-ca\") pod \"f73492d9-5852-498a-a6b7-10e2f446feeb\" (UID: \"f73492d9-5852-498a-a6b7-10e2f446feeb\") "
Oct 31 19:30:20.977780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.977736    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-kube-api-access-xwkvt\" (UniqueName: \"kubernetes.io/projected/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-kube-api-access-xwkvt\") pod \"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac\" (UID: \"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac\") "
Oct 31 19:30:20.977833 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.977785    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73492d9-5852-498a-a6b7-10e2f446feeb-kube-api-access-f9h9j\" (UniqueName: \"kubernetes.io/projected/f73492d9-5852-498a-a6b7-10e2f446feeb-kube-api-access-f9h9j\") pod \"f73492d9-5852-498a-a6b7-10e2f446feeb\" (UID: \"f73492d9-5852-498a-a6b7-10e2f446feeb\") "
Oct 31 19:30:20.977833 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.977783    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-maas-api-service-ca" pod "5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" (UID: "5bddf03c-e580-4a67-b122-2fb0ebb7b7ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:20.977959 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.977942    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73492d9-5852-498a-a6b7-10e2f446feeb-maas-api-service-ca" pod "f73492d9-5852-498a-a6b7-10e2f446feeb" (UID: "f73492d9-5852-498a-a6b7-10e2f446feeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:20.978279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.978260    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73492d9-5852-498a-a6b7-10e2f446feeb-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:20.978279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.978276    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:20.979990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.979966    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73492d9-5852-498a-a6b7-10e2f446feeb-kube-api-access-f9h9j" pod "f73492d9-5852-498a-a6b7-10e2f446feeb" (UID: "f73492d9-5852-498a-a6b7-10e2f446feeb"). InnerVolumeSpecName "kube-api-access-f9h9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:20.979990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:20.979972    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-kube-api-access-xwkvt" pod "5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" (UID: "5bddf03c-e580-4a67-b122-2fb0ebb7b7ac"). InnerVolumeSpecName "kube-api-access-xwkvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:21.079330 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.079298    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwkvt\" (UniqueName: \"kubernetes.io/projected/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac-kube-api-access-xwkvt\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:21.079330 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.079331    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9h9j\" (UniqueName: \"kubernetes.io/projected/f73492d9-5852-498a-a6b7-10e2f446feeb-kube-api-access-f9h9j\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:21.101618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.101590    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"]
Oct 31 19:30:21.115192 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.115169    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d22e36-29891250-lt52v"]
Oct 31 19:30:21.634135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.634097    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6" event={"ID":"5bddf03c-e580-4a67-b122-2fb0ebb7b7ac","Type":"ContainerDied","Data":"c0419ca80a0d31690b290352a9a4072b6a46910c82a057dc30d6a5d77b9adfe0"}
Oct 31 19:30:21.634678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.634172    4484 scope.go:122] "RemoveContainer" containerID="e35b9846cf32dbf72ef695ad60218114917367954a83f648611a0359b7fd8aca"
Oct 31 19:30:21.634678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.634119    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"
Oct 31 19:30:21.639830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.639808    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84" event={"ID":"f73492d9-5852-498a-a6b7-10e2f446feeb","Type":"ContainerDied","Data":"8ef29d86227f2a39498ee57293db56dbfaa3baffa323fa2b60c4bb657a49dc4e"}
Oct 31 19:30:21.639830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.639823    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"
Oct 31 19:30:21.645189 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.645168    4484 scope.go:122] "RemoveContainer" containerID="0f334596f21b21dfb29c6276165e1278daa6a9e30030da70b10abf434cf547ad"
Oct 31 19:30:21.790358 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.790263    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"
Oct 31 19:30:21.806762 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.804689    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"]
Oct 31 19:30:21.818755 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.818732    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-6279f5-29891250-vkm84"]
Oct 31 19:30:21.890347 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.890279    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c0f8eb1-6091-43f3-8928-9b06bad2daae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0f8eb1-6091-43f3-8928-9b06bad2daae-maas-api-service-ca\") pod \"7c0f8eb1-6091-43f3-8928-9b06bad2daae\" (UID: \"7c0f8eb1-6091-43f3-8928-9b06bad2daae\") "
Oct 31 19:30:21.890511 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.890423    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0f8eb1-6091-43f3-8928-9b06bad2daae-kube-api-access-67kvl\" (UniqueName: \"kubernetes.io/projected/7c0f8eb1-6091-43f3-8928-9b06bad2daae-kube-api-access-67kvl\") pod \"7c0f8eb1-6091-43f3-8928-9b06bad2daae\" (UID: \"7c0f8eb1-6091-43f3-8928-9b06bad2daae\") "
Oct 31 19:30:21.890646 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.890626    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0f8eb1-6091-43f3-8928-9b06bad2daae-maas-api-service-ca" pod "7c0f8eb1-6091-43f3-8928-9b06bad2daae" (UID: "7c0f8eb1-6091-43f3-8928-9b06bad2daae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:21.890816 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.890796    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0f8eb1-6091-43f3-8928-9b06bad2daae-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:21.892601 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.892582    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0f8eb1-6091-43f3-8928-9b06bad2daae-kube-api-access-67kvl" pod "7c0f8eb1-6091-43f3-8928-9b06bad2daae" (UID: "7c0f8eb1-6091-43f3-8928-9b06bad2daae"). InnerVolumeSpecName "kube-api-access-67kvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:21.927383 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.927354    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"]
Oct 31 19:30:21.931358 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.931333    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-88085e-29891250-nf5d6"]
Oct 31 19:30:21.992503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.992462    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67kvl\" (UniqueName: \"kubernetes.io/projected/7c0f8eb1-6091-43f3-8928-9b06bad2daae-kube-api-access-67kvl\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:21.994916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.994894    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" path="/var/lib/kubelet/pods/0f886bc5-ea49-4717-8e09-d3fe02c3d334/volumes"
Oct 31 19:30:21.995289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.995276    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5467e676-3261-4bb9-9ef4-b8a296072512" path="/var/lib/kubelet/pods/5467e676-3261-4bb9-9ef4-b8a296072512/volumes"
Oct 31 19:30:21.995714 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.995700    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" path="/var/lib/kubelet/pods/5bddf03c-e580-4a67-b122-2fb0ebb7b7ac/volumes"
Oct 31 19:30:21.996361 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.996346    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a901ad86-48e1-422b-a342-64e3433fd101" path="/var/lib/kubelet/pods/a901ad86-48e1-422b-a342-64e3433fd101/volumes"
Oct 31 19:30:21.996752 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:21.996738    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb" path="/var/lib/kubelet/pods/f73492d9-5852-498a-a6b7-10e2f446feeb/volumes"
Oct 31 19:30:22.205288 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:22.205156    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:22.649970 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:22.649935    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m" event={"ID":"7c0f8eb1-6091-43f3-8928-9b06bad2daae","Type":"ContainerDied","Data":"9df4058c24a8c0c5a2b29bc468e6067461df33fcc9300c976f08cff3783b016a"}
Oct 31 19:30:22.650412 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:22.649996    4484 scope.go:122] "RemoveContainer" containerID="702e2ed7a7f171f4ea296e6d52abe6933cc6095303bb5c65edf42c993f76d613"
Oct 31 19:30:22.650412 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:22.649955    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"
Oct 31 19:30:22.755092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:22.755059    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:22.819516 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:22.819455    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"]
Oct 31 19:30:22.824157 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:22.824130    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a8587d-29891250-nbd2m"]
Oct 31 19:30:23.580594 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:23.580556    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:23.994326 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:23.994296    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" path="/var/lib/kubelet/pods/7c0f8eb1-6091-43f3-8928-9b06bad2daae/volumes"
Oct 31 19:30:24.255400 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:24.255321    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:25.368119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:25.368086    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:25.481746 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:25.481715    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:30:26.144985 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:26.144948    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:29.228369 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:29.228336    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"]
Oct 31 19:30:29.336724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:29.336689    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"]
Oct 31 19:30:29.571324 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:29.571292    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:29.867215 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:29.867120    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:30.113403 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:30.113370    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:30.360937 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:30.360877    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:32.162425 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:32.162380    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:32.208803 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:32.208774    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:30:34.229120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.229072    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl" podUID="9cae31d6-d498-4abc-8151-95c59aeead05" containerName="payload-processing" containerID="cri-o://8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f" gracePeriod=25
Oct 31 19:30:34.337561 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.337463    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj" podUID="ad6a4ff5-d235-47ff-804c-314cfa54eb17" containerName="payload-pre-processing" containerID="cri-o://98c770b51f3a9f8c02b94e8bbb20b61adad4af68e8dec0eec94d493cc6b56b00" gracePeriod=25
Oct 31 19:30:34.751078 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.751024    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:34.759657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.759141    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"
Oct 31 19:30:34.820516 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.820291    4484 generic.go:356] "Generic (PLEG): container finished" podID="ad6a4ff5-d235-47ff-804c-314cfa54eb17" containerID="98c770b51f3a9f8c02b94e8bbb20b61adad4af68e8dec0eec94d493cc6b56b00" exitCode=0
Oct 31 19:30:34.820743 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.820589    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj" event={"ID":"ad6a4ff5-d235-47ff-804c-314cfa54eb17","Type":"ContainerDied","Data":"98c770b51f3a9f8c02b94e8bbb20b61adad4af68e8dec0eec94d493cc6b56b00"}
Oct 31 19:30:34.849979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.849941    4484 generic.go:356] "Generic (PLEG): container finished" podID="9cae31d6-d498-4abc-8151-95c59aeead05" containerID="8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f" exitCode=0
Oct 31 19:30:34.850159 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.850064    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl" event={"ID":"9cae31d6-d498-4abc-8151-95c59aeead05","Type":"ContainerDied","Data":"8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f"}
Oct 31 19:30:34.850159 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.850115    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl" event={"ID":"9cae31d6-d498-4abc-8151-95c59aeead05","Type":"ContainerDied","Data":"e215de9b4105b04379ee04bb8244087fadbcb169f4794fb3b635254022cd423c"}
Oct 31 19:30:34.850159 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.850145    4484 scope.go:122] "RemoveContainer" containerID="8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f"
Oct 31 19:30:34.850327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.850318    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"
Oct 31 19:30:34.855526 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.853823    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cae31d6-d498-4abc-8151-95c59aeead05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cae31d6-d498-4abc-8151-95c59aeead05-plugins-config-volume\") pod \"9cae31d6-d498-4abc-8151-95c59aeead05\" (UID: \"9cae31d6-d498-4abc-8151-95c59aeead05\") "
Oct 31 19:30:34.855526 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.853953    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cae31d6-d498-4abc-8151-95c59aeead05-kube-api-access-75wsr\" (UniqueName: \"kubernetes.io/projected/9cae31d6-d498-4abc-8151-95c59aeead05-kube-api-access-75wsr\") pod \"9cae31d6-d498-4abc-8151-95c59aeead05\" (UID: \"9cae31d6-d498-4abc-8151-95c59aeead05\") "
Oct 31 19:30:34.855526 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.855073    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cae31d6-d498-4abc-8151-95c59aeead05-plugins-config-volume" pod "9cae31d6-d498-4abc-8151-95c59aeead05" (UID: "9cae31d6-d498-4abc-8151-95c59aeead05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:34.926510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.925689    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cae31d6-d498-4abc-8151-95c59aeead05-kube-api-access-75wsr" pod "9cae31d6-d498-4abc-8151-95c59aeead05" (UID: "9cae31d6-d498-4abc-8151-95c59aeead05"). InnerVolumeSpecName "kube-api-access-75wsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:34.950688 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.950656    4484 scope.go:122] "RemoveContainer" containerID="8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f"
Oct 31 19:30:34.951136 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:34.951109    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f\": container with ID starting with 8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f not found: ID does not exist" containerID="8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f"
Oct 31 19:30:34.951264 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.951149    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f"} err="failed to get container status \"8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f\": rpc error: code = NotFound desc = could not find container \"8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f\": container with ID starting with 8a4b4ffc756cb0ba06beb9f29518647f2209437aa00495849809d5d22fee8c8f not found: ID does not exist"
Oct 31 19:30:34.955290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.955093    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cae31d6-d498-4abc-8151-95c59aeead05-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:34.955290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:34.955119    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75wsr\" (UniqueName: \"kubernetes.io/projected/9cae31d6-d498-4abc-8151-95c59aeead05-kube-api-access-75wsr\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:35.536819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.437977    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"]
Oct 31 19:30:35.537838 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.537809    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c249cd-59cbc8c7f-4cccl"]
Oct 31 19:30:35.568499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.566702    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"
Oct 31 19:30:35.744746 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.744664    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad6a4ff5-d235-47ff-804c-314cfa54eb17-kube-api-access-97mc5\" (UniqueName: \"kubernetes.io/projected/ad6a4ff5-d235-47ff-804c-314cfa54eb17-kube-api-access-97mc5\") pod \"ad6a4ff5-d235-47ff-804c-314cfa54eb17\" (UID: \"ad6a4ff5-d235-47ff-804c-314cfa54eb17\") "
Oct 31 19:30:35.745060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.745042    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad6a4ff5-d235-47ff-804c-314cfa54eb17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6a4ff5-d235-47ff-804c-314cfa54eb17-plugins-config-volume\") pod \"ad6a4ff5-d235-47ff-804c-314cfa54eb17\" (UID: \"ad6a4ff5-d235-47ff-804c-314cfa54eb17\") "
Oct 31 19:30:35.746068 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.746045    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad6a4ff5-d235-47ff-804c-314cfa54eb17-plugins-config-volume" pod "ad6a4ff5-d235-47ff-804c-314cfa54eb17" (UID: "ad6a4ff5-d235-47ff-804c-314cfa54eb17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:30:35.793505 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.791074    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6a4ff5-d235-47ff-804c-314cfa54eb17-kube-api-access-97mc5" pod "ad6a4ff5-d235-47ff-804c-314cfa54eb17" (UID: "ad6a4ff5-d235-47ff-804c-314cfa54eb17"). InnerVolumeSpecName "kube-api-access-97mc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:30:35.855501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.847932    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97mc5\" (UniqueName: \"kubernetes.io/projected/ad6a4ff5-d235-47ff-804c-314cfa54eb17-kube-api-access-97mc5\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:35.855501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.847960    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad6a4ff5-d235-47ff-804c-314cfa54eb17-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:30:35.934499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.934093    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:35.945391 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.945359    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj" event={"ID":"ad6a4ff5-d235-47ff-804c-314cfa54eb17","Type":"ContainerDied","Data":"aa9e800afbf6ca75fc49111e300c020f828e290e842069fbfb4276798e45d9cb"}
Oct 31 19:30:35.945661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.945370    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"
Oct 31 19:30:35.945799 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:35.945649    4484 scope.go:122] "RemoveContainer" containerID="98c770b51f3a9f8c02b94e8bbb20b61adad4af68e8dec0eec94d493cc6b56b00"
Oct 31 19:30:36.046267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:36.046173    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cae31d6-d498-4abc-8151-95c59aeead05" path="/var/lib/kubelet/pods/9cae31d6-d498-4abc-8151-95c59aeead05/volumes"
Oct 31 19:30:36.071499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:36.062568    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cwp95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:30:36.071499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:36.062638    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:30:36.127515 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:36.126449    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"]
Oct 31 19:30:36.176410 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:36.176376    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c249cd-84cb74567f-v5xlj"]
Oct 31 19:30:36.575169 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:36.575132    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:37.994764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:37.994730    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad6a4ff5-d235-47ff-804c-314cfa54eb17" path="/var/lib/kubelet/pods/ad6a4ff5-d235-47ff-804c-314cfa54eb17/volumes"
Oct 31 19:30:41.270906 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:41.270867    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:41.775890 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:41.775843    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:42.011340 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:42.011302    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:43.324984 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:43.324952    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:43.850060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:43.850018    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:44.992114 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:44.992053    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:45.243770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:45.243669    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:45.504691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:45.504601    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:46.318866 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:46.318826    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:51.136434 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:51.136395    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:51.384231 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:51.384196    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:51.658292 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:51.658261    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:51.961365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:51.961271    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:54.823650 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:54.823619    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:30:54.826978 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:54.826959    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:30:55.468616 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:30:55.468583    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:30:55.928608 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:55.928574    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:58.476561 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:58.476522    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:58.768285 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:30:58.768256    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:00.726349 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:00.726318    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:01.024874 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:01.024839    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:02.503993 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:02.503959    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:03.876305 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:03.876267    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:05.789875 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:05.789834    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:07.995290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:07.995258    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"]
Oct 31 19:31:08.347560 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:08.347440    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:08.590771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:08.590738    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:08.837932 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:08.837890    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:10.691865 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:10.691825    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:12.996023 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:12.995987    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck" podUID="a6e5f974-d0c6-4f66-8173-1aafbc948a40" containerName="payload-processing" containerID="cri-o://a9da4553f9148da24639609d360b9691f3c19e0ea611fe5efa0a6662d537a1af" gracePeriod=25
Oct 31 19:31:13.197580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:13.197463    4484 generic.go:356] "Generic (PLEG): container finished" podID="a6e5f974-d0c6-4f66-8173-1aafbc948a40" containerID="a9da4553f9148da24639609d360b9691f3c19e0ea611fe5efa0a6662d537a1af" exitCode=0
Oct 31 19:31:13.197971 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:13.197807    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck" event={"ID":"a6e5f974-d0c6-4f66-8173-1aafbc948a40","Type":"ContainerDied","Data":"a9da4553f9148da24639609d360b9691f3c19e0ea611fe5efa0a6662d537a1af"}
Oct 31 19:31:13.270376 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:13.270353    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"
Oct 31 19:31:13.386196 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:13.386162    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e5f974-d0c6-4f66-8173-1aafbc948a40-kube-api-access-2qk2k\" (UniqueName: \"kubernetes.io/projected/a6e5f974-d0c6-4f66-8173-1aafbc948a40-kube-api-access-2qk2k\") pod \"a6e5f974-d0c6-4f66-8173-1aafbc948a40\" (UID: \"a6e5f974-d0c6-4f66-8173-1aafbc948a40\") "
Oct 31 19:31:13.386394 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:13.386247    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e5f974-d0c6-4f66-8173-1aafbc948a40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e5f974-d0c6-4f66-8173-1aafbc948a40-plugins-config-volume\") pod \"a6e5f974-d0c6-4f66-8173-1aafbc948a40\" (UID: \"a6e5f974-d0c6-4f66-8173-1aafbc948a40\") "
Oct 31 19:31:13.386663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:13.386640    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e5f974-d0c6-4f66-8173-1aafbc948a40-plugins-config-volume" pod "a6e5f974-d0c6-4f66-8173-1aafbc948a40" (UID: "a6e5f974-d0c6-4f66-8173-1aafbc948a40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:31:13.386760 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:13.386740    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e5f974-d0c6-4f66-8173-1aafbc948a40-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:31:13.388754 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:13.388686    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e5f974-d0c6-4f66-8173-1aafbc948a40-kube-api-access-2qk2k" pod "a6e5f974-d0c6-4f66-8173-1aafbc948a40" (UID: "a6e5f974-d0c6-4f66-8173-1aafbc948a40"). InnerVolumeSpecName "kube-api-access-2qk2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:31:13.488050 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:13.488020    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qk2k\" (UniqueName: \"kubernetes.io/projected/a6e5f974-d0c6-4f66-8173-1aafbc948a40-kube-api-access-2qk2k\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:31:14.187808 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:14.187771    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:14.206611 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:14.206589    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"
Oct 31 19:31:14.206754 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:14.206619    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck" event={"ID":"a6e5f974-d0c6-4f66-8173-1aafbc948a40","Type":"ContainerDied","Data":"7b57e1f94f3be94710c85b24638d4268299d3f05a03eca7087d02ef755e83283"}
Oct 31 19:31:14.206754 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:14.206694    4484 scope.go:122] "RemoveContainer" containerID="a9da4553f9148da24639609d360b9691f3c19e0ea611fe5efa0a6662d537a1af"
Oct 31 19:31:14.374721 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:14.374647    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"]
Oct 31 19:31:14.425105 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:14.425067    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c97ce8-558b56bf5-d2cck"]
Oct 31 19:31:15.544437 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:15.544388    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:15.994026 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:15.993994    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e5f974-d0c6-4f66-8173-1aafbc948a40" path="/var/lib/kubelet/pods/a6e5f974-d0c6-4f66-8173-1aafbc948a40/volumes"
Oct 31 19:31:16.462296 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:16.462197    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:31:20.485972 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:20.485940    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:25.225030 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:25.224992    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:25.469307 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:31:25.469271    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:31:30.033546 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:30.033509    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:34.844044 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:34.844005    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:39.557963 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:39.557927    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:44.262316 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:44.262272    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:48.580673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:48.580626    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:49.095818 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:49.095783    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:49.381649 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:49.381553    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:53.840398 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:53.840364    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:55.470911 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:31:55.470870    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:31:56.499718 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:56.499683    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:58.544078 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:58.544037    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:59.488898 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:59.488859    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:59.901056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:31:59.900969    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:02.698655 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:02.698620    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:04.202084 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:04.202044    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:04.432725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:04.432686    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:04.679261 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:04.679229    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:06.035944 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:06.035865    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:06.576877 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:06.576844    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:07.343658 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:07.343620    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:07.741298 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:07.741257    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:08.705113 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:08.705081    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:09.449683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:09.449648    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:13.742177 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:13.742142    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:14.222007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:14.221915    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:14.519779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:14.519741    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:14.768412 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:14.768372    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"]
Oct 31 19:32:14.879325 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:14.879244    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"]
Oct 31 19:32:15.116766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:15.116733    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:15.370664 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:15.370628    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:18.351115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:18.351067    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:32:19.281493 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:19.281429    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:19.769421 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:19.769333    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk" podUID="7d4bd02a-689d-4f80-b2ff-1045c62dfa8a" containerName="payload-processing" containerID="cri-o://73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd" gracePeriod=25
Oct 31 19:32:19.880545 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:19.880459    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww" podUID="c63845cf-8d2c-4bc4-99e0-6576f10f924b" containerName="payload-pre-processing" containerID="cri-o://12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7" gracePeriod=25
Oct 31 19:32:20.092398 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.091980    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"
Oct 31 19:32:20.209098 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.209056    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-kube-api-access-dctng\" (UniqueName: \"kubernetes.io/projected/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-kube-api-access-dctng\") pod \"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a\" (UID: \"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a\") "
Oct 31 19:32:20.209319 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.209237    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-plugins-config-volume\") pod \"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a\" (UID: \"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a\") "
Oct 31 19:32:20.210253 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.210220    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-plugins-config-volume" pod "7d4bd02a-689d-4f80-b2ff-1045c62dfa8a" (UID: "7d4bd02a-689d-4f80-b2ff-1045c62dfa8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:32:20.225521 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.222062    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-kube-api-access-dctng" pod "7d4bd02a-689d-4f80-b2ff-1045c62dfa8a" (UID: "7d4bd02a-689d-4f80-b2ff-1045c62dfa8a"). InnerVolumeSpecName "kube-api-access-dctng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:32:20.250542 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.247885    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"
Oct 31 19:32:20.312635 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.312529    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63845cf-8d2c-4bc4-99e0-6576f10f924b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63845cf-8d2c-4bc4-99e0-6576f10f924b-plugins-config-volume\") pod \"c63845cf-8d2c-4bc4-99e0-6576f10f924b\" (UID: \"c63845cf-8d2c-4bc4-99e0-6576f10f924b\") "
Oct 31 19:32:20.312824 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.312687    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63845cf-8d2c-4bc4-99e0-6576f10f924b-kube-api-access-9rdgw\" (UniqueName: \"kubernetes.io/projected/c63845cf-8d2c-4bc4-99e0-6576f10f924b-kube-api-access-9rdgw\") pod \"c63845cf-8d2c-4bc4-99e0-6576f10f924b\" (UID: \"c63845cf-8d2c-4bc4-99e0-6576f10f924b\") "
Oct 31 19:32:20.312994 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.312956    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63845cf-8d2c-4bc4-99e0-6576f10f924b-plugins-config-volume" pod "c63845cf-8d2c-4bc4-99e0-6576f10f924b" (UID: "c63845cf-8d2c-4bc4-99e0-6576f10f924b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:32:20.313387 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.313361    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63845cf-8d2c-4bc4-99e0-6576f10f924b-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:20.313521 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.313391    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dctng\" (UniqueName: \"kubernetes.io/projected/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-kube-api-access-dctng\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:20.313521 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.313406    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:20.315944 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.315913    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63845cf-8d2c-4bc4-99e0-6576f10f924b-kube-api-access-9rdgw" pod "c63845cf-8d2c-4bc4-99e0-6576f10f924b" (UID: "c63845cf-8d2c-4bc4-99e0-6576f10f924b"). InnerVolumeSpecName "kube-api-access-9rdgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:32:20.414909 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.414868    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rdgw\" (UniqueName: \"kubernetes.io/projected/c63845cf-8d2c-4bc4-99e0-6576f10f924b-kube-api-access-9rdgw\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:32:20.477105 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.477055    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:20.727555 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.716991    4484 generic.go:356] "Generic (PLEG): container finished" podID="c63845cf-8d2c-4bc4-99e0-6576f10f924b" containerID="12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7" exitCode=0
Oct 31 19:32:20.727555 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.717105    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww" event={"ID":"c63845cf-8d2c-4bc4-99e0-6576f10f924b","Type":"ContainerDied","Data":"12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7"}
Oct 31 19:32:20.727555 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.717163    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww" event={"ID":"c63845cf-8d2c-4bc4-99e0-6576f10f924b","Type":"ContainerDied","Data":"7fb998941beb22b5cc5d88ea2dad57411447a302830920efa727a6872ae6cb3f"}
Oct 31 19:32:20.727555 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.717238    4484 scope.go:122] "RemoveContainer" containerID="12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7"
Oct 31 19:32:20.727555 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.717422    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"
Oct 31 19:32:20.736510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.733829    4484 generic.go:356] "Generic (PLEG): container finished" podID="7d4bd02a-689d-4f80-b2ff-1045c62dfa8a" containerID="73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd" exitCode=0
Oct 31 19:32:20.736510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.734016    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk" event={"ID":"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a","Type":"ContainerDied","Data":"73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd"}
Oct 31 19:32:20.736510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.734073    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk" event={"ID":"7d4bd02a-689d-4f80-b2ff-1045c62dfa8a","Type":"ContainerDied","Data":"12fe2f6c93fb2ba0b4f153e9fda8501ca54578edff2232be79657b28a97aca04"}
Oct 31 19:32:20.736510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.734152    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"
Oct 31 19:32:20.753868 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.753840    4484 scope.go:122] "RemoveContainer" containerID="12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7"
Oct 31 19:32:20.754224 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:32:20.754203    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7\": container with ID starting with 12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7 not found: ID does not exist" containerID="12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7"
Oct 31 19:32:20.754305 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.754232    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7"} err="failed to get container status \"12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7\": rpc error: code = NotFound desc = could not find container \"12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7\": container with ID starting with 12c52b4d11ceaa25cb08377b2e1cb3b219a8f8a79cfee40327c014faf8c6d0f7 not found: ID does not exist"
Oct 31 19:32:20.754305 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.754250    4484 scope.go:122] "RemoveContainer" containerID="73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd"
Oct 31 19:32:20.821582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.817033    4484 scope.go:122] "RemoveContainer" containerID="73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd"
Oct 31 19:32:20.821582 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:32:20.818137    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd\": container with ID starting with 73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd not found: ID does not exist" containerID="73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd"
Oct 31 19:32:20.821582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.818175    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd"} err="failed to get container status \"73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd\": rpc error: code = NotFound desc = could not find container \"73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd\": container with ID starting with 73b8e753a1f6ee2c5806c53830b3c574f394d8c10ed00885f05b09103a9ac5cd not found: ID does not exist"
Oct 31 19:32:20.920307 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.920261    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"]
Oct 31 19:32:20.953409 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:20.953368    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fdd317-75549c9b7-svqww"]
Oct 31 19:32:21.092444 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:21.092403    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"]
Oct 31 19:32:21.112605 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:21.112565    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fdd317-5975b54585-59lrk"]
Oct 31 19:32:22.039184 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:22.039146    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4bd02a-689d-4f80-b2ff-1045c62dfa8a" path="/var/lib/kubelet/pods/7d4bd02a-689d-4f80-b2ff-1045c62dfa8a/volumes"
Oct 31 19:32:22.039784 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:22.039757    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63845cf-8d2c-4bc4-99e0-6576f10f924b" path="/var/lib/kubelet/pods/c63845cf-8d2c-4bc4-99e0-6576f10f924b/volumes"
Oct 31 19:32:24.146622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:24.146583    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:25.469721 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:32:25.469685    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:32:28.842649 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:28.842611    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:33.534228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:33.534195    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:38.224092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:38.224054    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:43.011641 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:43.011594    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:47.770673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:47.770635    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:52.531163 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:52.531118    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:55.470923 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:32:55.470881    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:32:57.228408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:32:57.228366    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:02.015537 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:02.015499    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:04.527794 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:04.527759    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:06.723517 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:06.723454    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:09.035271 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:09.035234    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:11.419944 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:11.419903    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:16.129915 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:16.129876    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:17.577731 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:17.577696    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:20.330985 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:20.330944    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:22.112278 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:22.112241    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:23.516911 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:23.516870    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:25.469899 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:33:25.469858    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:33:26.817064 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:26.817012    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:28.175628 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:28.175592    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:28.565449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:28.565408    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:29.814318 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:29.814267    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:30.289568 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:30.289502    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:31.836139 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:31.836100    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:33.890828 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:33.890784    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:34.135319 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:34.135273    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:34.482141 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:34.482105    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:36.566189 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:36.566092    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:38.434880 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:38.434841    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:39.061907 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:39.061868    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:39.308936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:39.308901    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:41.472100 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:41.472061    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:42.872080 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:42.872019    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:43.150740 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:43.150648    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:46.112888 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:46.112853    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:46.440534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:46.440413    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:46.706886 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:46.706792    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:51.508709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:51.508671    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:54.642270 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:54.642238    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:33:55.468883 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:33:55.468848    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:33:56.277803 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:33:56.277752    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:01.229333 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:01.229298    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:04.902726 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:04.902687    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:06.041405 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:06.041359    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:09.420997 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:09.420955    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:10.929024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:10.928984    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:15.678675 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:15.678632    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:20.411767 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:20.411734    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:25.099885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:25.099849    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:25.471319 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:34:25.471232    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:34:29.883642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:29.883603    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:34.640047 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:34.640015    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:38.827493 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:38.827443    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:39.360467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:39.360432    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:44.215197 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:44.215157    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:48.143721 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:48.143680    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:48.407010 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:48.406915    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:48.931845 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:48.931813    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:49.248875 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:49.248840    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:51.046497 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:51.046437    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"]
Oct 31 19:34:51.097463 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:51.097423    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec5aa6ba-97vfb"]
Oct 31 19:34:51.994395 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:51.994362    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdbf6cd1-0704-4ecd-a16b-7e1d6230745a" path="/var/lib/kubelet/pods/fdbf6cd1-0704-4ecd-a16b-7e1d6230745a/volumes"
Oct 31 19:34:53.939347 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:53.939316    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:55.469830 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:34:55.469792    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:34:58.728462 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:34:58.728424    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:00.346116 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:00.346069    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:03.631869 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:03.631829    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:04.528886 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:04.528848    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:35:05.213270 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:05.213233    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:05.463976 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:05.463876    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:06.647059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:06.647011    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:08.438436 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:08.438397    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:08.688277 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:08.688237    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:09.823852 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:09.823818    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:13.224078 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:13.224034    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:15.385251 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:15.385222    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:18.034612 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:18.034582    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:21.408629 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:21.408586    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:22.663241 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:22.663204    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:25.470293 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:35:25.470254    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:35:27.366568 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:27.366527    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:32.005232 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:32.005185    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:36.233726 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:36.233688    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:35:36.682871 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:36.682835    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:41.446462 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:41.446430    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:46.255597 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:46.255563    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:51.015663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:51.015626    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:54.865406 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:54.865370    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:35:54.871393 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:54.871374    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:35:55.468721 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:35:55.468687    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:35:55.669914 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:55.669883    4484 scope.go:122] "RemoveContainer" containerID="932a90506a4ba929cefd50b82e1cbeb349a028bf5539f788ff38fe32a8cdbaa7"
Oct 31 19:35:55.812025 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:55.811994    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:57.939240 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:35:57.939206    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:00.538146 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:00.538105    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:05.247741 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:05.247704    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:05.544621 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:05.544544    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:05.956191 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:05.956103    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:06.841933 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:06.841900    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:07.188279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:07.188196    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:07.469333 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:07.469244    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:08.281576 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:08.281540    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:08.583852 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:08.583752    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:10.080307 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:10.080267    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:10.322313 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:10.322277    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:13.502844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:13.502789    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:15.011199 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:15.011163    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:18.968999 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:18.968966    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:36:19.732328 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:19.732293    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:24.449244 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:24.449206    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:25.469719 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:36:25.469680    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:36:29.122886 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:29.122846    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:31.193469 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:31.193433    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:32.216332 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:32.216296    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:32.587797 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:32.587763    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:33.936046 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:33.936016    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:36.987575 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:36.987460    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:37.329655 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:37.329563    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:38.945166 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:38.945133    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:40.284342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.284306    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:40.540747 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.540621    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:40.659357 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.659325    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"]
Oct 31 19:36:40.659966 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.659952    4484 cpu_manager.go:414] "containerMap: removing container" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" containerName="cleanup"
Oct 31 19:36:40.660016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.659967    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" containerName="cleanup"
Oct 31 19:36:40.660016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.659978    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a901ad86-48e1-422b-a342-64e3433fd101" containerName="cleanup"
Oct 31 19:36:40.660016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.659983    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a901ad86-48e1-422b-a342-64e3433fd101" containerName="cleanup"
Oct 31 19:36:40.660016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.659991    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" containerName="cleanup"
Oct 31 19:36:40.660016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.659995    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" containerName="cleanup"
Oct 31 19:36:40.660016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660004    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a901ad86-48e1-422b-a342-64e3433fd101" containerName="cleanup"
Oct 31 19:36:40.660016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660008    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a901ad86-48e1-422b-a342-64e3433fd101" containerName="cleanup"
Oct 31 19:36:40.660016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660015    4484 cpu_manager.go:414] "containerMap: removing container" podUID="39d07c93-21d5-47ad-b7f1-d29c35b31f57" containerName="cleanup"
Oct 31 19:36:40.660016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660019    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d07c93-21d5-47ad-b7f1-d29c35b31f57" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660027    4484 cpu_manager.go:414] "containerMap: removing container" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660031    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660039    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660043    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660056    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5467e676-3261-4bb9-9ef4-b8a296072512" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660060    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5467e676-3261-4bb9-9ef4-b8a296072512" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660068    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a6e5f974-d0c6-4f66-8173-1aafbc948a40" containerName="payload-processing"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660072    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e5f974-d0c6-4f66-8173-1aafbc948a40" containerName="payload-processing"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660080    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ad6a4ff5-d235-47ff-804c-314cfa54eb17" containerName="payload-pre-processing"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660084    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6a4ff5-d235-47ff-804c-314cfa54eb17" containerName="payload-pre-processing"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660091    4484 cpu_manager.go:414] "containerMap: removing container" podUID="39d07c93-21d5-47ad-b7f1-d29c35b31f57" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660095    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d07c93-21d5-47ad-b7f1-d29c35b31f57" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660105    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5467e676-3261-4bb9-9ef4-b8a296072512" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660109    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5467e676-3261-4bb9-9ef4-b8a296072512" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660115    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660119    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660126    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660130    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660139    4484 cpu_manager.go:414] "containerMap: removing container" podUID="9cae31d6-d498-4abc-8151-95c59aeead05" containerName="payload-processing"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660143    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cae31d6-d498-4abc-8151-95c59aeead05" containerName="payload-processing"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660149    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660153    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660161    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5467e676-3261-4bb9-9ef4-b8a296072512" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660165    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5467e676-3261-4bb9-9ef4-b8a296072512" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660178    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660182    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660190    4484 cpu_manager.go:414] "containerMap: removing container" podUID="7d4bd02a-689d-4f80-b2ff-1045c62dfa8a" containerName="payload-processing"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660194    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4bd02a-689d-4f80-b2ff-1045c62dfa8a" containerName="payload-processing"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660204    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660208    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660214    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660218    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660225    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660229    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660238    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660243    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" containerName="cleanup"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660250    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c63845cf-8d2c-4bc4-99e0-6576f10f924b" containerName="payload-pre-processing"
Oct 31 19:36:40.660291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660255    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63845cf-8d2c-4bc4-99e0-6576f10f924b" containerName="payload-pre-processing"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660342    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a901ad86-48e1-422b-a342-64e3433fd101" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660350    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d07c93-21d5-47ad-b7f1-d29c35b31f57" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660356    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660361    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660367    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660373    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660380    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5467e676-3261-4bb9-9ef4-b8a296072512" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660384    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e5f974-d0c6-4f66-8173-1aafbc948a40" containerName="payload-processing"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660389    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6a4ff5-d235-47ff-804c-314cfa54eb17" containerName="payload-pre-processing"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660396    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d07c93-21d5-47ad-b7f1-d29c35b31f57" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660401    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5467e676-3261-4bb9-9ef4-b8a296072512" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660406    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660412    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cae31d6-d498-4abc-8151-95c59aeead05" containerName="payload-processing"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660416    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660421    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660427    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5467e676-3261-4bb9-9ef4-b8a296072512" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660433    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660439    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4bd02a-689d-4f80-b2ff-1045c62dfa8a" containerName="payload-processing"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660444    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660449    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f886bc5-ea49-4717-8e09-d3fe02c3d334" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660455    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660461    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba38b8bd-6dcb-4b68-9226-2455456bcf74" containerName="cleanup"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660466    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63845cf-8d2c-4bc4-99e0-6576f10f924b" containerName="payload-pre-processing"
Oct 31 19:36:40.661503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.660491    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a901ad86-48e1-422b-a342-64e3433fd101" containerName="cleanup"
Oct 31 19:36:40.664011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.663996    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"
Oct 31 19:36:40.773497 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.750767    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"]
Oct 31 19:36:40.775204 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.775169    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4daaf71-c968-4d2a-8a18-9542333ac2c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8\" (UID: \"e4daaf71-c968-4d2a-8a18-9542333ac2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"
Oct 31 19:36:40.775342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.775237    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnqm7\" (UniqueName: \"kubernetes.io/projected/e4daaf71-c968-4d2a-8a18-9542333ac2c7-kube-api-access-rnqm7\") pod \"payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8\" (UID: \"e4daaf71-c968-4d2a-8a18-9542333ac2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"
Oct 31 19:36:40.787855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.787823    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-0f794f\""
Oct 31 19:36:40.787989 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.787901    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-0f794f-dockercfg-ng4dt\""
Oct 31 19:36:40.876316 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.876238    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4daaf71-c968-4d2a-8a18-9542333ac2c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8\" (UID: \"e4daaf71-c968-4d2a-8a18-9542333ac2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"
Oct 31 19:36:40.876316 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.876294    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnqm7\" (UniqueName: \"kubernetes.io/projected/e4daaf71-c968-4d2a-8a18-9542333ac2c7-kube-api-access-rnqm7\") pod \"payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8\" (UID: \"e4daaf71-c968-4d2a-8a18-9542333ac2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"
Oct 31 19:36:40.876981 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.876961    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4daaf71-c968-4d2a-8a18-9542333ac2c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8\" (UID: \"e4daaf71-c968-4d2a-8a18-9542333ac2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"
Oct 31 19:36:40.934861 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.934836    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnqm7\" (UniqueName: \"kubernetes.io/projected/e4daaf71-c968-4d2a-8a18-9542333ac2c7-kube-api-access-rnqm7\") pod \"payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8\" (UID: \"e4daaf71-c968-4d2a-8a18-9542333ac2c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"
Oct 31 19:36:40.977590 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:40.977557    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"
Oct 31 19:36:41.353910 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:41.353883    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"]
Oct 31 19:36:41.354656 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:36:41.354632    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4daaf71_c968_4d2a_8a18_9542333ac2c7.slice/crio-6907accf002c2c5c5690a113850a8f264f4c4918cb208cb3141216447a3e3eb3 WatchSource:0}: Error finding container 6907accf002c2c5c5690a113850a8f264f4c4918cb208cb3141216447a3e3eb3: Status 404 returned error can't find the container with id 6907accf002c2c5c5690a113850a8f264f4c4918cb208cb3141216447a3e3eb3
Oct 31 19:36:41.429806 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:41.429778    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8" event={"ID":"e4daaf71-c968-4d2a-8a18-9542333ac2c7","Type":"ContainerStarted","Data":"6907accf002c2c5c5690a113850a8f264f4c4918cb208cb3141216447a3e3eb3"}
Oct 31 19:36:42.437288 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:42.437252    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8" event={"ID":"e4daaf71-c968-4d2a-8a18-9542333ac2c7","Type":"ContainerStarted","Data":"529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d"}
Oct 31 19:36:42.437823 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:42.437464    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"
Oct 31 19:36:42.554043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:42.553989    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8" podStartSLOduration=2.553974958 podStartE2EDuration="2.553974958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:42.552643822 +0000 UTC m=+4548.699093686" watchObservedRunningTime="2026-10-31 19:36:42.553974958 +0000 UTC m=+4548.700424820"
Oct 31 19:36:43.735676 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:43.735636    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:46.177018 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:46.176976    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:48.434837 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:48.434803    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:52.578325 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:52.578284    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:52.876563 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:52.876465    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"]
Oct 31 19:36:52.986510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:52.986463    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"]
Oct 31 19:36:53.233177 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:53.233084    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:53.448182 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:53.448157    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"
Oct 31 19:36:53.475077 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:53.475018    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:55.470626 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:36:55.470587    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:36:57.877548 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:57.877506    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq" podUID="5410d4d5-bc30-456d-a17b-65e13b084937" containerName="payload-processing" containerID="cri-o://0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212" gracePeriod=25
Oct 31 19:36:57.986818 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:57.986769    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2" podUID="b3a0d3d9-238f-44e4-9894-11feed845de8" containerName="payload-pre-processing" containerID="cri-o://6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36" gracePeriod=25
Oct 31 19:36:58.255901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:58.255862    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:58.511753 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:36:58.511659    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:02.904936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:02.904903    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:05.838058 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:05.838018    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq" podUID="5410d4d5-bc30-456d-a17b-65e13b084937" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:37:05.838502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:05.838021    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2" podUID="b3a0d3d9-238f-44e4-9894-11feed845de8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:37:05.891518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:05.891463    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:07.635328 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:07.635287    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:10.407750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:10.407713    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:12.343737 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:12.343687    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:15.838431 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:15.838394    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2" podUID="b3a0d3d9-238f-44e4-9894-11feed845de8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:37:15.838892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:15.838395    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq" podUID="5410d4d5-bc30-456d-a17b-65e13b084937" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:37:17.142330 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:17.142298    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:18.155998 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.155670    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"
Oct 31 19:37:18.221388 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.221353    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5410d4d5-bc30-456d-a17b-65e13b084937-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5410d4d5-bc30-456d-a17b-65e13b084937-plugins-config-volume\") pod \"5410d4d5-bc30-456d-a17b-65e13b084937\" (UID: \"5410d4d5-bc30-456d-a17b-65e13b084937\") "
Oct 31 19:37:18.221640 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.221432    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5410d4d5-bc30-456d-a17b-65e13b084937-kube-api-access-xzqs7\" (UniqueName: \"kubernetes.io/projected/5410d4d5-bc30-456d-a17b-65e13b084937-kube-api-access-xzqs7\") pod \"5410d4d5-bc30-456d-a17b-65e13b084937\" (UID: \"5410d4d5-bc30-456d-a17b-65e13b084937\") "
Oct 31 19:37:18.221845 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.221820    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5410d4d5-bc30-456d-a17b-65e13b084937-plugins-config-volume" pod "5410d4d5-bc30-456d-a17b-65e13b084937" (UID: "5410d4d5-bc30-456d-a17b-65e13b084937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:37:18.224343 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.224313    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5410d4d5-bc30-456d-a17b-65e13b084937-kube-api-access-xzqs7" pod "5410d4d5-bc30-456d-a17b-65e13b084937" (UID: "5410d4d5-bc30-456d-a17b-65e13b084937"). InnerVolumeSpecName "kube-api-access-xzqs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:37:18.245885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.245860    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"
Oct 31 19:37:18.322860 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.322824    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a0d3d9-238f-44e4-9894-11feed845de8-kube-api-access-vhsgv\" (UniqueName: \"kubernetes.io/projected/b3a0d3d9-238f-44e4-9894-11feed845de8-kube-api-access-vhsgv\") pod \"b3a0d3d9-238f-44e4-9894-11feed845de8\" (UID: \"b3a0d3d9-238f-44e4-9894-11feed845de8\") "
Oct 31 19:37:18.323059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.322950    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a0d3d9-238f-44e4-9894-11feed845de8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a0d3d9-238f-44e4-9894-11feed845de8-plugins-config-volume\") pod \"b3a0d3d9-238f-44e4-9894-11feed845de8\" (UID: \"b3a0d3d9-238f-44e4-9894-11feed845de8\") "
Oct 31 19:37:18.323372 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.323345    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a0d3d9-238f-44e4-9894-11feed845de8-plugins-config-volume" pod "b3a0d3d9-238f-44e4-9894-11feed845de8" (UID: "b3a0d3d9-238f-44e4-9894-11feed845de8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:37:18.323372 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.323359    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5410d4d5-bc30-456d-a17b-65e13b084937-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:18.323614 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.323391    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzqs7\" (UniqueName: \"kubernetes.io/projected/5410d4d5-bc30-456d-a17b-65e13b084937-kube-api-access-xzqs7\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:18.325192 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.325173    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a0d3d9-238f-44e4-9894-11feed845de8-kube-api-access-vhsgv" pod "b3a0d3d9-238f-44e4-9894-11feed845de8" (UID: "b3a0d3d9-238f-44e4-9894-11feed845de8"). InnerVolumeSpecName "kube-api-access-vhsgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:37:18.424589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.424465    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhsgv\" (UniqueName: \"kubernetes.io/projected/b3a0d3d9-238f-44e4-9894-11feed845de8-kube-api-access-vhsgv\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:18.424589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.424547    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3a0d3d9-238f-44e4-9894-11feed845de8-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:37:18.670637 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.670608    4484 generic.go:356] "Generic (PLEG): container finished" podID="b3a0d3d9-238f-44e4-9894-11feed845de8" containerID="6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36" exitCode=0
Oct 31 19:37:18.670838 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.670650    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"
Oct 31 19:37:18.670838 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.670697    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2" event={"ID":"b3a0d3d9-238f-44e4-9894-11feed845de8","Type":"ContainerDied","Data":"6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36"}
Oct 31 19:37:18.670838 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.670750    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2" event={"ID":"b3a0d3d9-238f-44e4-9894-11feed845de8","Type":"ContainerDied","Data":"ce9f2bf67c7930b8a42b6494f54e4e6e2efdfed3071a1fdf8f7a3fa36ac7571a"}
Oct 31 19:37:18.670838 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.670773    4484 scope.go:122] "RemoveContainer" containerID="6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36"
Oct 31 19:37:18.672208 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.672176    4484 generic.go:356] "Generic (PLEG): container finished" podID="5410d4d5-bc30-456d-a17b-65e13b084937" containerID="0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212" exitCode=0
Oct 31 19:37:18.672535 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.672222    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"
Oct 31 19:37:18.672535 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.672233    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq" event={"ID":"5410d4d5-bc30-456d-a17b-65e13b084937","Type":"ContainerDied","Data":"0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212"}
Oct 31 19:37:18.672535 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.672263    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq" event={"ID":"5410d4d5-bc30-456d-a17b-65e13b084937","Type":"ContainerDied","Data":"faf113f1bfc568c4a376955a111046a5bedc7ead54e234a4ab1212f0ca7b22c1"}
Oct 31 19:37:18.680984 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.680968    4484 scope.go:122] "RemoveContainer" containerID="6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36"
Oct 31 19:37:18.681251 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:37:18.681232    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36\": container with ID starting with 6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36 not found: ID does not exist" containerID="6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36"
Oct 31 19:37:18.681325 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.681258    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36"} err="failed to get container status \"6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36\": rpc error: code = NotFound desc = could not find container \"6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36\": container with ID starting with 6027ca8e8f17fd06cc9f2b9622d6a84bed0d3591a91a22e223ea30a17f11ea36 not found: ID does not exist"
Oct 31 19:37:18.681325 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.681274    4484 scope.go:122] "RemoveContainer" containerID="0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212"
Oct 31 19:37:18.690163 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.690149    4484 scope.go:122] "RemoveContainer" containerID="0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212"
Oct 31 19:37:18.690432 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:37:18.690413    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212\": container with ID starting with 0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212 not found: ID does not exist" containerID="0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212"
Oct 31 19:37:18.690507 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.690441    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212"} err="failed to get container status \"0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212\": rpc error: code = NotFound desc = could not find container \"0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212\": container with ID starting with 0e1c7ce885c0b0057eea6633a36d4a2c5aa07a43fcb4415869e803bcf60dc212 not found: ID does not exist"
Oct 31 19:37:18.845150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.845114    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"]
Oct 31 19:37:18.853384 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.853358    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-d8f34e-75fccc887-hkkz2"]
Oct 31 19:37:18.964768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.964691    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"]
Oct 31 19:37:18.968970 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:18.968944    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-d8f34e-74b88d8df7-zfxrq"]
Oct 31 19:37:19.994162 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:19.994130    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5410d4d5-bc30-456d-a17b-65e13b084937" path="/var/lib/kubelet/pods/5410d4d5-bc30-456d-a17b-65e13b084937/volumes"
Oct 31 19:37:19.994653 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:19.994634    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a0d3d9-238f-44e4-9894-11feed845de8" path="/var/lib/kubelet/pods/b3a0d3d9-238f-44e4-9894-11feed845de8/volumes"
Oct 31 19:37:21.820049 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:21.820021    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:24.766613 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:24.766570    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:25.469594 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:37:25.469556    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:37:26.725077 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:26.725038    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:31.414394 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:31.414354    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:36.109198 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:36.109158    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:38.233845 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:38.233799    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:38.650821 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:38.650788    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:40.192307 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:40.192263    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:41.505296 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:41.505254    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:41.793779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:41.793681    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:42.045396 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:42.045314    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:44.905340 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:44.905298    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:45.838348 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:45.838314    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:46.102006 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:46.101909    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:46.699215 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:46.699169    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:46.976906 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:46.976806    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:48.329235 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:48.329199    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:48.607263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:48.607167    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:49.864104 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:49.864064    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:50.220005 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:50.219920    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:50.559862 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:50.559830    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:51.192561 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:51.192520    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:51.438709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:51.438674    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:52.086926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:52.086887    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:55.469605 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:37:55.469566    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:37:55.852245 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:55.852211    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:37:56.958248 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:56.958217    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:57.260943 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:57.260897    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:59.269258 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:37:59.269198    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:01.255191 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:01.255157    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:02.229607 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:02.229568    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:05.519652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:05.519618    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:06.212847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:06.212816    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:06.516378 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:06.516344    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:07.016650 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:07.016612    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:10.758184 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:10.758147    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:11.772413 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:11.772376    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:16.530495 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:16.530439    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:21.483947 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:21.483911    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:25.468758 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:38:25.468723    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:38:25.770275 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:25.770239    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:27.718893 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:27.718855    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:32.681375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:32.681339    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:36.396724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:36.396690    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:36.726238 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:36.726152    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:37.760398 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:37.760359    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:38.182253 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:38.182167    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:38.797065 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:38.797027    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:39.035381 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:39.035349    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:38:39.512142 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:39.512109    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:43.082280 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:43.082239    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:45.901043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:45.901003    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:46.182654 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:46.182557    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"]
Oct 31 19:38:46.291788 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:46.291758    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"]
Oct 31 19:38:46.550876 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:46.550837    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:46.872433 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:46.872342    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:48.177842 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:48.177805    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:51.183689 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:51.183646    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq" podUID="169ec18f-05f8-4fec-a540-9ade4b14863d" containerName="payload-processing" containerID="cri-o://6dd354e23edafee9517db07e36a0c29735df9ed94ae817d56898ea0bb5b9c25c" gracePeriod=25
Oct 31 19:38:51.292561 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:51.292523    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx" podUID="c010d33f-79f2-4396-83aa-ef85d00d4c81" containerName="payload-pre-processing" containerID="cri-o://d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22" gracePeriod=25
Oct 31 19:38:52.546082 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:52.546046    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:53.289299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:53.289251    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:55.468550 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:38:55.468515    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:38:57.935355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:57.935318    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx" podUID="c010d33f-79f2-4396-83aa-ef85d00d4c81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:38:57.935837 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:57.935319    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq" podUID="169ec18f-05f8-4fec-a540-9ade4b14863d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:38:58.135079 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:38:58.135043    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:03.065999 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:03.065960    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:05.319591 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:05.319557    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:05.768327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:05.768299    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:06.402870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:06.402836    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:06.888267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:06.888207    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:07.686698 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:07.686661    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:07.935592 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:07.935558    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq" podUID="169ec18f-05f8-4fec-a540-9ade4b14863d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:07.935767 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:07.935559    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx" podUID="c010d33f-79f2-4396-83aa-ef85d00d4c81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:09.570638 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:09.570582    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:10.842033 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:10.841972    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:11.138620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.138513    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:11.395039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.394948    4484 generic.go:356] "Generic (PLEG): container finished" podID="169ec18f-05f8-4fec-a540-9ade4b14863d" containerID="6dd354e23edafee9517db07e36a0c29735df9ed94ae817d56898ea0bb5b9c25c" exitCode=0
Oct 31 19:39:11.395039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.395026    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq" event={"ID":"169ec18f-05f8-4fec-a540-9ade4b14863d","Type":"ContainerDied","Data":"6dd354e23edafee9517db07e36a0c29735df9ed94ae817d56898ea0bb5b9c25c"}
Oct 31 19:39:11.455682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.455656    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"
Oct 31 19:39:11.578441 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.578409    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"
Oct 31 19:39:11.593518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.593454    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169ec18f-05f8-4fec-a540-9ade4b14863d-kube-api-access-qd9lf\" (UniqueName: \"kubernetes.io/projected/169ec18f-05f8-4fec-a540-9ade4b14863d-kube-api-access-qd9lf\") pod \"169ec18f-05f8-4fec-a540-9ade4b14863d\" (UID: \"169ec18f-05f8-4fec-a540-9ade4b14863d\") "
Oct 31 19:39:11.593686 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.593641    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169ec18f-05f8-4fec-a540-9ade4b14863d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169ec18f-05f8-4fec-a540-9ade4b14863d-plugins-config-volume\") pod \"169ec18f-05f8-4fec-a540-9ade4b14863d\" (UID: \"169ec18f-05f8-4fec-a540-9ade4b14863d\") "
Oct 31 19:39:11.593999 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.593975    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169ec18f-05f8-4fec-a540-9ade4b14863d-plugins-config-volume" pod "169ec18f-05f8-4fec-a540-9ade4b14863d" (UID: "169ec18f-05f8-4fec-a540-9ade4b14863d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:11.594228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.594207    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169ec18f-05f8-4fec-a540-9ade4b14863d-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:11.595644 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.595621    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169ec18f-05f8-4fec-a540-9ade4b14863d-kube-api-access-qd9lf" pod "169ec18f-05f8-4fec-a540-9ade4b14863d" (UID: "169ec18f-05f8-4fec-a540-9ade4b14863d"). InnerVolumeSpecName "kube-api-access-qd9lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:11.694944 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.694872    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c010d33f-79f2-4396-83aa-ef85d00d4c81-kube-api-access-rr68g\" (UniqueName: \"kubernetes.io/projected/c010d33f-79f2-4396-83aa-ef85d00d4c81-kube-api-access-rr68g\") pod \"c010d33f-79f2-4396-83aa-ef85d00d4c81\" (UID: \"c010d33f-79f2-4396-83aa-ef85d00d4c81\") "
Oct 31 19:39:11.695093 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.694977    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c010d33f-79f2-4396-83aa-ef85d00d4c81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c010d33f-79f2-4396-83aa-ef85d00d4c81-plugins-config-volume\") pod \"c010d33f-79f2-4396-83aa-ef85d00d4c81\" (UID: \"c010d33f-79f2-4396-83aa-ef85d00d4c81\") "
Oct 31 19:39:11.695438 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.695416    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c010d33f-79f2-4396-83aa-ef85d00d4c81-plugins-config-volume" pod "c010d33f-79f2-4396-83aa-ef85d00d4c81" (UID: "c010d33f-79f2-4396-83aa-ef85d00d4c81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:11.695521 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.695507    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd9lf\" (UniqueName: \"kubernetes.io/projected/169ec18f-05f8-4fec-a540-9ade4b14863d-kube-api-access-qd9lf\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:11.697419 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.697394    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c010d33f-79f2-4396-83aa-ef85d00d4c81-kube-api-access-rr68g" pod "c010d33f-79f2-4396-83aa-ef85d00d4c81" (UID: "c010d33f-79f2-4396-83aa-ef85d00d4c81"). InnerVolumeSpecName "kube-api-access-rr68g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:11.797015 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.796978    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr68g\" (UniqueName: \"kubernetes.io/projected/c010d33f-79f2-4396-83aa-ef85d00d4c81-kube-api-access-rr68g\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:11.797015 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:11.797008    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c010d33f-79f2-4396-83aa-ef85d00d4c81-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:12.162293 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.162258    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:39:12.213185 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.213154    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:12.424401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.424320    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"
Oct 31 19:39:12.424401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.424361    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq" event={"ID":"169ec18f-05f8-4fec-a540-9ade4b14863d","Type":"ContainerDied","Data":"404c0ef1ff7324eed2255cef18f8b45385f138fe70eb427a7ee1cd11e15444da"}
Oct 31 19:39:12.424679 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.424435    4484 scope.go:122] "RemoveContainer" containerID="6dd354e23edafee9517db07e36a0c29735df9ed94ae817d56898ea0bb5b9c25c"
Oct 31 19:39:12.426785 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.426758    4484 generic.go:356] "Generic (PLEG): container finished" podID="c010d33f-79f2-4396-83aa-ef85d00d4c81" containerID="d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22" exitCode=0
Oct 31 19:39:12.427061 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.427032    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx" event={"ID":"c010d33f-79f2-4396-83aa-ef85d00d4c81","Type":"ContainerDied","Data":"d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22"}
Oct 31 19:39:12.427164 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.427086    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx" event={"ID":"c010d33f-79f2-4396-83aa-ef85d00d4c81","Type":"ContainerDied","Data":"94ef2fba7847eacb37c16dd3ee5f53be16c2306cf9f31403cab14728de1fc759"}
Oct 31 19:39:12.427164 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.427157    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"
Oct 31 19:39:12.437153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.437132    4484 scope.go:122] "RemoveContainer" containerID="d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22"
Oct 31 19:39:12.446721 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.446700    4484 scope.go:122] "RemoveContainer" containerID="d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22"
Oct 31 19:39:12.446998 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:39:12.446977    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22\": container with ID starting with d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22 not found: ID does not exist" containerID="d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22"
Oct 31 19:39:12.447059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.447007    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22"} err="failed to get container status \"d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22\": rpc error: code = NotFound desc = could not find container \"d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22\": container with ID starting with d6b9223e93e7f4945c9c83850a8b8e454d639e3163a6e5a34a8d2801bfe4ef22 not found: ID does not exist"
Oct 31 19:39:12.598918 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.598691    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"]
Oct 31 19:39:12.666535 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.662263    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bbd8c6-5fdcf8cf55-rqdnx"]
Oct 31 19:39:12.671430 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.671387    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"]
Oct 31 19:39:12.671810 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.671774    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerName="istio-proxy" containerID="cri-o://6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8" gracePeriod=30
Oct 31 19:39:12.691883 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.685983    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:12.724445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.724409    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"]
Oct 31 19:39:12.728113 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:12.728090    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bbd8c6-7467bf744f-ftscq"]
Oct 31 19:39:13.178775 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:13.178743    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:13.995424 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:13.995391    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169ec18f-05f8-4fec-a540-9ade4b14863d" path="/var/lib/kubelet/pods/169ec18f-05f8-4fec-a540-9ade4b14863d/volumes"
Oct 31 19:39:13.995883 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:13.995868    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c010d33f-79f2-4396-83aa-ef85d00d4c81" path="/var/lib/kubelet/pods/c010d33f-79f2-4396-83aa-ef85d00d4c81/volumes"
Oct 31 19:39:14.313890 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:14.313797    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:14.584049 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:14.583977    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:14.826992 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:14.826949    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:16.292143 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:16.292108    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:17.785950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:17.785912    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:18.036794 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.036740    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:39:18.041211 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.041187    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:18.177877 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.177844    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"]
Oct 31 19:39:18.191248 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191210    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-envoy\") pod \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") "
Oct 31 19:39:18.191409 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191263    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-socket\") pod \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") "
Oct 31 19:39:18.191409 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191294    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-credential-socket\") pod \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") "
Oct 31 19:39:18.191409 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191395    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-data\") pod \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") "
Oct 31 19:39:18.191582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191444    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-podinfo\") pod \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") "
Oct 31 19:39:18.191582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191505    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-token\" (UniqueName: \"kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-token\") pod \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") "
Oct 31 19:39:18.191703 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191598    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-certs\") pod \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") "
Oct 31 19:39:18.191703 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191684    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-kube-api-access-kv2wq\" (UniqueName: \"kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-kube-api-access-kv2wq\") pod \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") "
Oct 31 19:39:18.191815 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191602    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-credential-socket" pod "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" (UID: "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:18.191815 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191731    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istiod-ca-cert\") pod \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\" (UID: \"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7\") "
Oct 31 19:39:18.191815 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191619    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-socket" pod "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" (UID: "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:18.191815 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191736    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-data" pod "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" (UID: "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:18.191815 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.191800    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-certs" pod "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" (UID: "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:18.192265 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.192242    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istiod-ca-cert" pod "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" (UID: "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:18.192351 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.192253    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:18.192351 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.192290    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:18.192351 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.192310    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:18.192351 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.192324    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:18.193782 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.193755    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-envoy" pod "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" (UID: "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:18.194081 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.194060    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-token" pod "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" (UID: "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:18.194141 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.194093    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-kube-api-access-kv2wq" pod "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" (UID: "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7"). InnerVolumeSpecName "kube-api-access-kv2wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:18.194247 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.194227    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-podinfo" pod "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" (UID: "b435e8f9-377a-4d33-bee0-a5e03e4eb4d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:39:18.294119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.294029    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:18.294119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.294060    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:18.294119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.294074    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv2wq\" (UniqueName: \"kubernetes.io/projected/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-kube-api-access-kv2wq\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:18.294119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.294086    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:18.294119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.294098    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:18.399786 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.399751    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:18.467749 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.467714    4484 generic.go:356] "Generic (PLEG): container finished" podID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerID="6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8" exitCode=0
Oct 31 19:39:18.467948 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.467771    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"
Oct 31 19:39:18.467948 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.467789    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" event={"ID":"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7","Type":"ContainerDied","Data":"6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8"}
Oct 31 19:39:18.467948 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.467840    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f" event={"ID":"b435e8f9-377a-4d33-bee0-a5e03e4eb4d7","Type":"ContainerDied","Data":"c16437a194ff89a558b4e8469cc27dc43490b555a78bee44b2797abd83e40724"}
Oct 31 19:39:18.467948 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.467863    4484 scope.go:122] "RemoveContainer" containerID="6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8"
Oct 31 19:39:18.478600 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.478579    4484 scope.go:122] "RemoveContainer" containerID="8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2"
Oct 31 19:39:18.487909 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.487885    4484 scope.go:122] "RemoveContainer" containerID="6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8"
Oct 31 19:39:18.488193 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:39:18.488166    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8\": container with ID starting with 6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8 not found: ID does not exist" containerID="6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8"
Oct 31 19:39:18.488283 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.488204    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8"} err="failed to get container status \"6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8\": rpc error: code = NotFound desc = could not find container \"6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8\": container with ID starting with 6d87a95df250f7c744d2d7c28b8d741a1dcd8dc8230e576d48fbee1b1094d7c8 not found: ID does not exist"
Oct 31 19:39:18.488283 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.488222    4484 scope.go:122] "RemoveContainer" containerID="8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2"
Oct 31 19:39:18.488507 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:39:18.488464    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2\": container with ID starting with 8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2 not found: ID does not exist" containerID="8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2"
Oct 31 19:39:18.488507 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.488496    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2"} err="failed to get container status \"8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2\": rpc error: code = NotFound desc = could not find container \"8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2\": container with ID starting with 8c6b7b310ac4fdb57de1d2d6ca1501d0a09d74515a01b9d863427bd951e0a9e2 not found: ID does not exist"
Oct 31 19:39:18.639274 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.639193    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"]
Oct 31 19:39:18.646038 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:18.646012    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c4731a-openshift-default-84cc4f786c-xf45f"]
Oct 31 19:39:19.520624 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:19.520569    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:19.995589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:19.995561    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" path="/var/lib/kubelet/pods/b435e8f9-377a-4d33-bee0-a5e03e4eb4d7/volumes"
Oct 31 19:39:20.004000 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:20.003977    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:20.808626 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:20.808593    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:21.503784 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:21.503744    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:24.453235 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:24.453192    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:25.471038 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:39:25.471004    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:39:26.333153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:26.333121    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:27.960368 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:27.960339    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:28.075718 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:28.075685    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"]
Oct 31 19:39:28.184698 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:28.184670    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"]
Oct 31 19:39:28.427198 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:28.427159    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:28.707853 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:28.707743    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:30.468866 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:30.468829    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"]
Oct 31 19:39:30.552753 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:30.552717    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:30.576420 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:30.576381    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"]
Oct 31 19:39:30.852850 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:30.852813    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:31.100131 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:31.100089    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:31.654582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:31.654552    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:32.258695 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:32.258660    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"]
Oct 31 19:39:32.258920 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:32.258887    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy" containerID="cri-o://8b5ce1ad1e47d015773a24cdd8ab9144363f8880ed3d1727b25c3ec79ff45808" gracePeriod=30
Oct 31 19:39:32.309454 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:32.309421    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"]
Oct 31 19:39:32.309685 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:32.309661    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy" containerID="cri-o://7991af225a961b3c7d6afe314eb12a1e06f6d64be83d81aef42d7cc1e2de72d7" gracePeriod=30
Oct 31 19:39:33.076425 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.076382    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66" podUID="1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a" containerName="payload-processing" containerID="cri-o://a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522" gracePeriod=25
Oct 31 19:39:33.185120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.185073    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg" podUID="0e5e6ede-697c-47c4-b53b-99a417db3f2d" containerName="payload-pre-processing" containerID="cri-o://774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab" gracePeriod=25
Oct 31 19:39:33.200287 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.200252    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6" podUID="a91dc442-830a-4436-bab0-fedc94f75038" containerName="main" containerID="cri-o://04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce" gracePeriod=45
Oct 31 19:39:33.231405 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.231373    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6" podUID="a91dc442-830a-4436-bab0-fedc94f75038" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": dial tcp 10.133.0.64:8000: connect: connection refused"
Oct 31 19:39:33.475922 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.475896    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 19:39:33.564336 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.564301    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-home\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-home\") pod \"a91dc442-830a-4436-bab0-fedc94f75038\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") "
Oct 31 19:39:33.564543 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.564445    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-tmp-dir\") pod \"a91dc442-830a-4436-bab0-fedc94f75038\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") "
Oct 31 19:39:33.564543 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.564512    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a91dc442-830a-4436-bab0-fedc94f75038-tls-certs\" (UniqueName: \"kubernetes.io/secret/a91dc442-830a-4436-bab0-fedc94f75038-tls-certs\") pod \"a91dc442-830a-4436-bab0-fedc94f75038\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") "
Oct 31 19:39:33.564654 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.564616    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91dc442-830a-4436-bab0-fedc94f75038-kube-api-access-4r2ff\" (UniqueName: \"kubernetes.io/projected/a91dc442-830a-4436-bab0-fedc94f75038-kube-api-access-4r2ff\") pod \"a91dc442-830a-4436-bab0-fedc94f75038\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") "
Oct 31 19:39:33.564702 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.564679    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-model-cache\") pod \"a91dc442-830a-4436-bab0-fedc94f75038\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") "
Oct 31 19:39:33.564870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.564742    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-dshm\") pod \"a91dc442-830a-4436-bab0-fedc94f75038\" (UID: \"a91dc442-830a-4436-bab0-fedc94f75038\") "
Oct 31 19:39:33.565026 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.565003    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-tmp-dir" pod "a91dc442-830a-4436-bab0-fedc94f75038" (UID: "a91dc442-830a-4436-bab0-fedc94f75038"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:33.565318 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.565288    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:33.565959 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.565930    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-model-cache" pod "a91dc442-830a-4436-bab0-fedc94f75038" (UID: "a91dc442-830a-4436-bab0-fedc94f75038"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:33.566332 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.566296    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-home" pod "a91dc442-830a-4436-bab0-fedc94f75038" (UID: "a91dc442-830a-4436-bab0-fedc94f75038"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:33.566615 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.566582    4484 generic.go:356] "Generic (PLEG): container finished" podID="a91dc442-830a-4436-bab0-fedc94f75038" containerID="04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce" exitCode=0
Oct 31 19:39:33.566709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.566636    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"
Oct 31 19:39:33.566709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.566664    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6" event={"ID":"a91dc442-830a-4436-bab0-fedc94f75038","Type":"ContainerDied","Data":"04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce"}
Oct 31 19:39:33.566709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.566703    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6" event={"ID":"a91dc442-830a-4436-bab0-fedc94f75038","Type":"ContainerDied","Data":"393b148f1f838dff58d9a6fb506dfad70c68459cc712cc1cf0be0c7aa0f6fd01"}
Oct 31 19:39:33.566874 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.566725    4484 scope.go:122] "RemoveContainer" containerID="04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce"
Oct 31 19:39:33.571559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.571537    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-dshm" pod "a91dc442-830a-4436-bab0-fedc94f75038" (UID: "a91dc442-830a-4436-bab0-fedc94f75038"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:33.571884 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.571850    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91dc442-830a-4436-bab0-fedc94f75038-tls-certs" pod "a91dc442-830a-4436-bab0-fedc94f75038" (UID: "a91dc442-830a-4436-bab0-fedc94f75038"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:39:33.572119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.572096    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91dc442-830a-4436-bab0-fedc94f75038-kube-api-access-4r2ff" pod "a91dc442-830a-4436-bab0-fedc94f75038" (UID: "a91dc442-830a-4436-bab0-fedc94f75038"). InnerVolumeSpecName "kube-api-access-4r2ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:33.597209 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.597178    4484 scope.go:122] "RemoveContainer" containerID="04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce"
Oct 31 19:39:33.597667 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:39:33.597634    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce\": container with ID starting with 04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce not found: ID does not exist" containerID="04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce"
Oct 31 19:39:33.597795 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.597675    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce"} err="failed to get container status \"04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce\": rpc error: code = NotFound desc = could not find container \"04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce\": container with ID starting with 04bb59fc277638e09fc90b61dace31167ee9e3aab7707761ec4d305689b523ce not found: ID does not exist"
Oct 31 19:39:33.666180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.666091    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91dc442-830a-4436-bab0-fedc94f75038-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:33.666180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.666122    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r2ff\" (UniqueName: \"kubernetes.io/projected/a91dc442-830a-4436-bab0-fedc94f75038-kube-api-access-4r2ff\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:33.666180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.666138    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:33.666180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.666150    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:33.666180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.666161    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91dc442-830a-4436-bab0-fedc94f75038-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:33.996742 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:33.996716    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"]
Oct 31 19:39:34.040746 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:34.040716    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a8587d/simulator-w0-a8587d-kserve-f887488bb-d62z6"]
Oct 31 19:39:34.709107 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:34.709062    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:35.010365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:35.010330    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-cwp95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:39:35.010584 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:35.010399    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:39:35.470060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:35.469957    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk" podUID="73b43632-5dbb-4491-b1b8-48c1383f7266" containerName="payload-processing" containerID="cri-o://9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a" gracePeriod=25
Oct 31 19:39:35.576740 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:35.576683    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg" podUID="65891533-3ba8-44da-b413-d6936580f382" containerName="payload-pre-processing" containerID="cri-o://8aaf1882962d417e01bc9eb10b9956b6ca5f99d9c5156515f80044d97c7f0203" gracePeriod=25
Oct 31 19:39:35.993984 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:35.993950    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91dc442-830a-4436-bab0-fedc94f75038" path="/var/lib/kubelet/pods/a91dc442-830a-4436-bab0-fedc94f75038/volumes"
Oct 31 19:39:36.324906 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:36.324813    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk" podUID="73b43632-5dbb-4491-b1b8-48c1383f7266" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:36.325073 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:36.324926    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg" podUID="65891533-3ba8-44da-b413-d6936580f382" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:36.683712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:36.683636    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66" podUID="1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:36.683868 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:36.683634    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg" podUID="0e5e6ede-697c-47c4-b53b-99a417db3f2d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:36.822275 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:36.822223    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:37.139267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.139228    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:37.264192 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.264152    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2gq6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:39:37.264359 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.264215    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:39:37.616982 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.616939    4484 generic.go:356] "Generic (PLEG): container finished" podID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerID="8b5ce1ad1e47d015773a24cdd8ab9144363f8880ed3d1727b25c3ec79ff45808" exitCode=0
Oct 31 19:39:37.617160 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.617071    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" event={"ID":"0d62a86e-a9fd-403b-92c8-b9617f803b85","Type":"ContainerDied","Data":"8b5ce1ad1e47d015773a24cdd8ab9144363f8880ed3d1727b25c3ec79ff45808"}
Oct 31 19:39:37.617160 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.617146    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95" event={"ID":"0d62a86e-a9fd-403b-92c8-b9617f803b85","Type":"ContainerDied","Data":"44ff0b08c33d13d012290603ff84175f6376dfa31cc11363b04c7eb33f7d0d32"}
Oct 31 19:39:37.617300 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.617174    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44ff0b08c33d13d012290603ff84175f6376dfa31cc11363b04c7eb33f7d0d32"
Oct 31 19:39:37.621392 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.620437    4484 generic.go:356] "Generic (PLEG): container finished" podID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerID="7991af225a961b3c7d6afe314eb12a1e06f6d64be83d81aef42d7cc1e2de72d7" exitCode=0
Oct 31 19:39:37.621392 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.620519    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" event={"ID":"0cbcf347-7edf-499d-87ea-4ea645f2631e","Type":"ContainerDied","Data":"7991af225a961b3c7d6afe314eb12a1e06f6d64be83d81aef42d7cc1e2de72d7"}
Oct 31 19:39:37.621726 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.621710    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:39:37.671865 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.671799    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:39:37.713005 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.712960    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-socket\") pod \"0cbcf347-7edf-499d-87ea-4ea645f2631e\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") "
Oct 31 19:39:37.713204 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713077    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-certs\") pod \"0cbcf347-7edf-499d-87ea-4ea645f2631e\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") "
Oct 31 19:39:37.713204 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713137    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-credential-socket\") pod \"0cbcf347-7edf-499d-87ea-4ea645f2631e\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") "
Oct 31 19:39:37.713204 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713175    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-envoy\") pod \"0d62a86e-a9fd-403b-92c8-b9617f803b85\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") "
Oct 31 19:39:37.713408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713208    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-podinfo\") pod \"0cbcf347-7edf-499d-87ea-4ea645f2631e\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") "
Oct 31 19:39:37.713408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713246    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-kube-api-access-dk9t4\" (UniqueName: \"kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-kube-api-access-dk9t4\") pod \"0cbcf347-7edf-499d-87ea-4ea645f2631e\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") "
Oct 31 19:39:37.713408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713321    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbcf347-7edf-499d-87ea-4ea645f2631e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbcf347-7edf-499d-87ea-4ea645f2631e-istiod-ca-cert\") pod \"0cbcf347-7edf-499d-87ea-4ea645f2631e\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") "
Oct 31 19:39:37.713408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713363    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-kube-api-access-bnfmg\" (UniqueName: \"kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-kube-api-access-bnfmg\") pod \"0d62a86e-a9fd-403b-92c8-b9617f803b85\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") "
Oct 31 19:39:37.713700 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713422    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-envoy\") pod \"0cbcf347-7edf-499d-87ea-4ea645f2631e\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") "
Oct 31 19:39:37.713700 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713514    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-data\") pod \"0cbcf347-7edf-499d-87ea-4ea645f2631e\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") "
Oct 31 19:39:37.713700 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713563    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-data\") pod \"0d62a86e-a9fd-403b-92c8-b9617f803b85\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") "
Oct 31 19:39:37.713700 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713600    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d62a86e-a9fd-403b-92c8-b9617f803b85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d62a86e-a9fd-403b-92c8-b9617f803b85-istiod-ca-cert\") pod \"0d62a86e-a9fd-403b-92c8-b9617f803b85\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") "
Oct 31 19:39:37.713700 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713629    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-credential-socket\") pod \"0d62a86e-a9fd-403b-92c8-b9617f803b85\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") "
Oct 31 19:39:37.713700 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713663    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-socket\") pod \"0d62a86e-a9fd-403b-92c8-b9617f803b85\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") "
Oct 31 19:39:37.714020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713703    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-token\" (UniqueName: \"kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-token\") pod \"0cbcf347-7edf-499d-87ea-4ea645f2631e\" (UID: \"0cbcf347-7edf-499d-87ea-4ea645f2631e\") "
Oct 31 19:39:37.714020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713731    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-certs\") pod \"0d62a86e-a9fd-403b-92c8-b9617f803b85\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") "
Oct 31 19:39:37.714020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713768    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-podinfo\") pod \"0d62a86e-a9fd-403b-92c8-b9617f803b85\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") "
Oct 31 19:39:37.714020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.713813    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-token\" (UniqueName: \"kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-token\") pod \"0d62a86e-a9fd-403b-92c8-b9617f803b85\" (UID: \"0d62a86e-a9fd-403b-92c8-b9617f803b85\") "
Oct 31 19:39:37.715217 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.714992    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-credential-socket" pod "0d62a86e-a9fd-403b-92c8-b9617f803b85" (UID: "0d62a86e-a9fd-403b-92c8-b9617f803b85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.715554 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.715403    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-data" pod "0cbcf347-7edf-499d-87ea-4ea645f2631e" (UID: "0cbcf347-7edf-499d-87ea-4ea645f2631e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.715664 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.715576    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-socket" pod "0d62a86e-a9fd-403b-92c8-b9617f803b85" (UID: "0d62a86e-a9fd-403b-92c8-b9617f803b85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.715741 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.715681    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-certs" pod "0d62a86e-a9fd-403b-92c8-b9617f803b85" (UID: "0d62a86e-a9fd-403b-92c8-b9617f803b85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.716770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.715957    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-certs" pod "0cbcf347-7edf-499d-87ea-4ea645f2631e" (UID: "0cbcf347-7edf-499d-87ea-4ea645f2631e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.716770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.716016    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-data" pod "0d62a86e-a9fd-403b-92c8-b9617f803b85" (UID: "0d62a86e-a9fd-403b-92c8-b9617f803b85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.716770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.716022    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-credential-socket" pod "0cbcf347-7edf-499d-87ea-4ea645f2631e" (UID: "0cbcf347-7edf-499d-87ea-4ea645f2631e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.716770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.716486    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-socket" pod "0cbcf347-7edf-499d-87ea-4ea645f2631e" (UID: "0cbcf347-7edf-499d-87ea-4ea645f2631e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.716770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.716729    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbcf347-7edf-499d-87ea-4ea645f2631e-istiod-ca-cert" pod "0cbcf347-7edf-499d-87ea-4ea645f2631e" (UID: "0cbcf347-7edf-499d-87ea-4ea645f2631e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:37.718713 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.718677    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-token" pod "0d62a86e-a9fd-403b-92c8-b9617f803b85" (UID: "0d62a86e-a9fd-403b-92c8-b9617f803b85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:37.718927 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.718897    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-token" pod "0cbcf347-7edf-499d-87ea-4ea645f2631e" (UID: "0cbcf347-7edf-499d-87ea-4ea645f2631e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:37.723193 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.723157    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-kube-api-access-bnfmg" pod "0d62a86e-a9fd-403b-92c8-b9617f803b85" (UID: "0d62a86e-a9fd-403b-92c8-b9617f803b85"). InnerVolumeSpecName "kube-api-access-bnfmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:37.723391 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.723361    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-podinfo" pod "0cbcf347-7edf-499d-87ea-4ea645f2631e" (UID: "0cbcf347-7edf-499d-87ea-4ea645f2631e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:39:37.724026 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.723995    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-envoy" pod "0d62a86e-a9fd-403b-92c8-b9617f803b85" (UID: "0d62a86e-a9fd-403b-92c8-b9617f803b85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.724273 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.724244    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-kube-api-access-dk9t4" pod "0cbcf347-7edf-499d-87ea-4ea645f2631e" (UID: "0cbcf347-7edf-499d-87ea-4ea645f2631e"). InnerVolumeSpecName "kube-api-access-dk9t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:37.724691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.724667    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d62a86e-a9fd-403b-92c8-b9617f803b85-istiod-ca-cert" pod "0d62a86e-a9fd-403b-92c8-b9617f803b85" (UID: "0d62a86e-a9fd-403b-92c8-b9617f803b85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:37.724924 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.724903    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-envoy" pod "0cbcf347-7edf-499d-87ea-4ea645f2631e" (UID: "0cbcf347-7edf-499d-87ea-4ea645f2631e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:37.729110 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.729084    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-podinfo" pod "0d62a86e-a9fd-403b-92c8-b9617f803b85" (UID: "0d62a86e-a9fd-403b-92c8-b9617f803b85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:39:37.815525 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815459    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815525 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815519    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815525 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815534    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815547    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk9t4\" (UniqueName: \"kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-kube-api-access-dk9t4\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815564    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbcf347-7edf-499d-87ea-4ea645f2631e-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815577    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnfmg\" (UniqueName: \"kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-kube-api-access-bnfmg\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815590    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815603    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815617    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815629    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d62a86e-a9fd-403b-92c8-b9617f803b85-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815641    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815654    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815669    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbcf347-7edf-499d-87ea-4ea645f2631e-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815683    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d62a86e-a9fd-403b-92c8-b9617f803b85-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815696    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815709    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d62a86e-a9fd-403b-92c8-b9617f803b85-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815721    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:37.815847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:37.815735    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbcf347-7edf-499d-87ea-4ea645f2631e-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:38.627382 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:38.627301    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"
Oct 31 19:39:38.627382 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:38.627333    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t" event={"ID":"0cbcf347-7edf-499d-87ea-4ea645f2631e","Type":"ContainerDied","Data":"4abd5e356820ab6069039c68988be129aafc67e76e5fe5ed373aca51ba5c17a1"}
Oct 31 19:39:38.627934 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:38.627395    4484 scope.go:122] "RemoveContainer" containerID="7991af225a961b3c7d6afe314eb12a1e06f6d64be83d81aef42d7cc1e2de72d7"
Oct 31 19:39:38.627934 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:38.627463    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"
Oct 31 19:39:38.848989 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:38.848955    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"]
Oct 31 19:39:38.853242 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:38.853214    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2gq6t"]
Oct 31 19:39:38.964509 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:38.964420    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"]
Oct 31 19:39:38.973587 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:38.973562    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-cwp95"]
Oct 31 19:39:39.994193 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:39.994152    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" path="/var/lib/kubelet/pods/0cbcf347-7edf-499d-87ea-4ea645f2631e/volumes"
Oct 31 19:39:39.994804 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:39.994784    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" path="/var/lib/kubelet/pods/0d62a86e-a9fd-403b-92c8-b9617f803b85/volumes"
Oct 31 19:39:41.929066 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:41.929035    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:43.152837 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:43.152804    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:45.335031 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:45.334995    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:46.323983 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:46.323939    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg" podUID="65891533-3ba8-44da-b413-d6936580f382" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:46.324199 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:46.324002    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk" podUID="73b43632-5dbb-4491-b1b8-48c1383f7266" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:46.683973 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:46.683888    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg" podUID="0e5e6ede-697c-47c4-b53b-99a417db3f2d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:46.684352 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:46.683887    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66" podUID="1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:46.697917 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:46.697882    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:50.883673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:50.883632    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:51.248256 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:51.248218    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:51.504650 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:51.504564    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:53.356591 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.356565    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"
Oct 31 19:39:53.450625 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.450598    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"
Oct 31 19:39:53.491890 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.491855    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-kube-api-access-9sc4l\" (UniqueName: \"kubernetes.io/projected/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-kube-api-access-9sc4l\") pod \"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a\" (UID: \"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a\") "
Oct 31 19:39:53.492071 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.491949    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-plugins-config-volume\") pod \"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a\" (UID: \"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a\") "
Oct 31 19:39:53.492286 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.492267    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-plugins-config-volume" pod "1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a" (UID: "1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:53.492544 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.492515    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:53.494337 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.494308    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-kube-api-access-9sc4l" pod "1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a" (UID: "1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a"). InnerVolumeSpecName "kube-api-access-9sc4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:53.593239 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.593163    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e5e6ede-697c-47c4-b53b-99a417db3f2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5e6ede-697c-47c4-b53b-99a417db3f2d-plugins-config-volume\") pod \"0e5e6ede-697c-47c4-b53b-99a417db3f2d\" (UID: \"0e5e6ede-697c-47c4-b53b-99a417db3f2d\") "
Oct 31 19:39:53.593400 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.593376    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5e6ede-697c-47c4-b53b-99a417db3f2d-kube-api-access-vs7ld\" (UniqueName: \"kubernetes.io/projected/0e5e6ede-697c-47c4-b53b-99a417db3f2d-kube-api-access-vs7ld\") pod \"0e5e6ede-697c-47c4-b53b-99a417db3f2d\" (UID: \"0e5e6ede-697c-47c4-b53b-99a417db3f2d\") "
Oct 31 19:39:53.593647 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.593619    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5e6ede-697c-47c4-b53b-99a417db3f2d-plugins-config-volume" pod "0e5e6ede-697c-47c4-b53b-99a417db3f2d" (UID: "0e5e6ede-697c-47c4-b53b-99a417db3f2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:53.593961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.593907    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sc4l\" (UniqueName: \"kubernetes.io/projected/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a-kube-api-access-9sc4l\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:53.593961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.593936    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e5e6ede-697c-47c4-b53b-99a417db3f2d-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:53.595988 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.595965    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5e6ede-697c-47c4-b53b-99a417db3f2d-kube-api-access-vs7ld" pod "0e5e6ede-697c-47c4-b53b-99a417db3f2d" (UID: "0e5e6ede-697c-47c4-b53b-99a417db3f2d"). InnerVolumeSpecName "kube-api-access-vs7ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:53.695423 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.695391    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs7ld\" (UniqueName: \"kubernetes.io/projected/0e5e6ede-697c-47c4-b53b-99a417db3f2d-kube-api-access-vs7ld\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:53.729660 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.729631    4484 generic.go:356] "Generic (PLEG): container finished" podID="1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a" containerID="a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522" exitCode=0
Oct 31 19:39:53.729814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.729675    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"
Oct 31 19:39:53.729814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.729717    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66" event={"ID":"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a","Type":"ContainerDied","Data":"a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522"}
Oct 31 19:39:53.729814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.729775    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66" event={"ID":"1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a","Type":"ContainerDied","Data":"22c7d0dc6437dfc0755267cd7d5238d65ee5d15aa2b456091e9a87d436de1f95"}
Oct 31 19:39:53.729814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.729809    4484 scope.go:122] "RemoveContainer" containerID="a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522"
Oct 31 19:39:53.731113 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.731091    4484 generic.go:356] "Generic (PLEG): container finished" podID="0e5e6ede-697c-47c4-b53b-99a417db3f2d" containerID="774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab" exitCode=0
Oct 31 19:39:53.731244 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.731123    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg" event={"ID":"0e5e6ede-697c-47c4-b53b-99a417db3f2d","Type":"ContainerDied","Data":"774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab"}
Oct 31 19:39:53.731244 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.731142    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"
Oct 31 19:39:53.731244 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.731177    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg" event={"ID":"0e5e6ede-697c-47c4-b53b-99a417db3f2d","Type":"ContainerDied","Data":"c0a3fa4f78125151723824da8ed774556b2581caa4c00234621b1e64e3a21166"}
Oct 31 19:39:53.740301 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.740193    4484 scope.go:122] "RemoveContainer" containerID="a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522"
Oct 31 19:39:53.740529 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:39:53.740512    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522\": container with ID starting with a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522 not found: ID does not exist" containerID="a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522"
Oct 31 19:39:53.740602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.740539    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522"} err="failed to get container status \"a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522\": rpc error: code = NotFound desc = could not find container \"a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522\": container with ID starting with a24fc0832b0eb503d2f67e1b5d06a55653e130e7af8f3bac515dc4058071c522 not found: ID does not exist"
Oct 31 19:39:53.740602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.740555    4484 scope.go:122] "RemoveContainer" containerID="774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab"
Oct 31 19:39:53.749277 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.749257    4484 scope.go:122] "RemoveContainer" containerID="774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab"
Oct 31 19:39:53.749551 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:39:53.749534    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab\": container with ID starting with 774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab not found: ID does not exist" containerID="774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab"
Oct 31 19:39:53.749609 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.749558    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab"} err="failed to get container status \"774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab\": rpc error: code = NotFound desc = could not find container \"774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab\": container with ID starting with 774ee6f3a65a6a8d6a2cce2efa74b0f15c921bda327a3b4f52585d61f63466ab not found: ID does not exist"
Oct 31 19:39:53.918317 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.918241    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"]
Oct 31 19:39:53.922192 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.922163    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d22e36-7486fccc5c-z8c66"]
Oct 31 19:39:53.993814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:53.993783    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a" path="/var/lib/kubelet/pods/1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a/volumes"
Oct 31 19:39:54.034441 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:54.034410    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"]
Oct 31 19:39:54.037666 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:54.037637    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d22e36-76bdf6fc99-bh2hg"]
Oct 31 19:39:55.244550 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.244518    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:55.470239 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:39:55.470200    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:39:55.498830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.498769    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:55.741407 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.741333    4484 scope.go:122] "RemoveContainer" containerID="8b5ce1ad1e47d015773a24cdd8ab9144363f8880ed3d1727b25c3ec79ff45808"
Oct 31 19:39:55.746119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.745910    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"
Oct 31 19:39:55.747315 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.747290    4484 generic.go:356] "Generic (PLEG): container finished" podID="65891533-3ba8-44da-b413-d6936580f382" containerID="8aaf1882962d417e01bc9eb10b9956b6ca5f99d9c5156515f80044d97c7f0203" exitCode=0
Oct 31 19:39:55.747421 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.747330    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg" event={"ID":"65891533-3ba8-44da-b413-d6936580f382","Type":"ContainerDied","Data":"8aaf1882962d417e01bc9eb10b9956b6ca5f99d9c5156515f80044d97c7f0203"}
Oct 31 19:39:55.749111 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.749058    4484 generic.go:356] "Generic (PLEG): container finished" podID="73b43632-5dbb-4491-b1b8-48c1383f7266" containerID="9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a" exitCode=0
Oct 31 19:39:55.749111 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.749113    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"
Oct 31 19:39:55.749300 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.749119    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk" event={"ID":"73b43632-5dbb-4491-b1b8-48c1383f7266","Type":"ContainerDied","Data":"9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a"}
Oct 31 19:39:55.749300 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.749159    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk" event={"ID":"73b43632-5dbb-4491-b1b8-48c1383f7266","Type":"ContainerDied","Data":"b25d7d7772d9f8dc146b88d3200b9c8187f866d8342e387ed7a0b9cf36842cb0"}
Oct 31 19:39:55.749300 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.749189    4484 scope.go:122] "RemoveContainer" containerID="9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a"
Oct 31 19:39:55.754096 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.754084    4484 scope.go:122] "RemoveContainer" containerID="9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a"
Oct 31 19:39:55.768749 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:39:55.768639    4484 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_payload-processing_payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk_openshift-ingress_73b43632-5dbb-4491-b1b8-48c1383f7266_0 in pod sandbox b25d7d7772d9f8dc146b88d3200b9c8187f866d8342e387ed7a0b9cf36842cb0 from index: no such id: '9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a'" containerID="9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a"
Oct 31 19:39:55.768749 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.768690    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a"} err="rpc error: code = Unknown desc = failed to delete container k8s_payload-processing_payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk_openshift-ingress_73b43632-5dbb-4491-b1b8-48c1383f7266_0 in pod sandbox b25d7d7772d9f8dc146b88d3200b9c8187f866d8342e387ed7a0b9cf36842cb0 from index: no such id: '9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a'"
Oct 31 19:39:55.768749 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.768716    4484 scope.go:122] "RemoveContainer" containerID="9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a"
Oct 31 19:39:55.769375 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:39:55.769327    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a\": container with ID starting with 9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a not found: ID does not exist" containerID="9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a"
Oct 31 19:39:55.769511 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.769387    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a"} err="failed to get container status \"9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a\": rpc error: code = NotFound desc = could not find container \"9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a\": container with ID starting with 9271b21a5c3ba953d3279de05c7df3b30b7d37314c1b4cbd4ad9d3ee5dc3400a not found: ID does not exist"
Oct 31 19:39:55.851396 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.851371    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"
Oct 31 19:39:55.921197 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.921167    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b43632-5dbb-4491-b1b8-48c1383f7266-kube-api-access-6ccd7\" (UniqueName: \"kubernetes.io/projected/73b43632-5dbb-4491-b1b8-48c1383f7266-kube-api-access-6ccd7\") pod \"73b43632-5dbb-4491-b1b8-48c1383f7266\" (UID: \"73b43632-5dbb-4491-b1b8-48c1383f7266\") "
Oct 31 19:39:55.921401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.921299    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b43632-5dbb-4491-b1b8-48c1383f7266-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b43632-5dbb-4491-b1b8-48c1383f7266-plugins-config-volume\") pod \"73b43632-5dbb-4491-b1b8-48c1383f7266\" (UID: \"73b43632-5dbb-4491-b1b8-48c1383f7266\") "
Oct 31 19:39:55.921918 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.921893    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b43632-5dbb-4491-b1b8-48c1383f7266-plugins-config-volume" pod "73b43632-5dbb-4491-b1b8-48c1383f7266" (UID: "73b43632-5dbb-4491-b1b8-48c1383f7266"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:55.923867 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.923848    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b43632-5dbb-4491-b1b8-48c1383f7266-kube-api-access-6ccd7" pod "73b43632-5dbb-4491-b1b8-48c1383f7266" (UID: "73b43632-5dbb-4491-b1b8-48c1383f7266"). InnerVolumeSpecName "kube-api-access-6ccd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:55.993787 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:55.993757    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e5e6ede-697c-47c4-b53b-99a417db3f2d" path="/var/lib/kubelet/pods/0e5e6ede-697c-47c4-b53b-99a417db3f2d/volumes"
Oct 31 19:39:56.022268 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.022240    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65891533-3ba8-44da-b413-d6936580f382-kube-api-access-l8sxs\" (UniqueName: \"kubernetes.io/projected/65891533-3ba8-44da-b413-d6936580f382-kube-api-access-l8sxs\") pod \"65891533-3ba8-44da-b413-d6936580f382\" (UID: \"65891533-3ba8-44da-b413-d6936580f382\") "
Oct 31 19:39:56.022409 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.022340    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65891533-3ba8-44da-b413-d6936580f382-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65891533-3ba8-44da-b413-d6936580f382-plugins-config-volume\") pod \"65891533-3ba8-44da-b413-d6936580f382\" (UID: \"65891533-3ba8-44da-b413-d6936580f382\") "
Oct 31 19:39:56.022694 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.022676    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ccd7\" (UniqueName: \"kubernetes.io/projected/73b43632-5dbb-4491-b1b8-48c1383f7266-kube-api-access-6ccd7\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:56.022768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.022690    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65891533-3ba8-44da-b413-d6936580f382-plugins-config-volume" pod "65891533-3ba8-44da-b413-d6936580f382" (UID: "65891533-3ba8-44da-b413-d6936580f382"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:56.022768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.022699    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b43632-5dbb-4491-b1b8-48c1383f7266-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:56.024537 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.024520    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65891533-3ba8-44da-b413-d6936580f382-kube-api-access-l8sxs" pod "65891533-3ba8-44da-b413-d6936580f382" (UID: "65891533-3ba8-44da-b413-d6936580f382"). InnerVolumeSpecName "kube-api-access-l8sxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:56.123620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.123583    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65891533-3ba8-44da-b413-d6936580f382-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:56.123620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.123610    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8sxs\" (UniqueName: \"kubernetes.io/projected/65891533-3ba8-44da-b413-d6936580f382-kube-api-access-l8sxs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:39:56.191231 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.191204    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"]
Oct 31 19:39:56.203348 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.203321    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a8587d-b69d589d9-zdfgk"]
Oct 31 19:39:56.347203 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.347106    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:56.680264 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.680182    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:56.755850 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.755820    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg" event={"ID":"65891533-3ba8-44da-b413-d6936580f382","Type":"ContainerDied","Data":"009325fe699fb633e58bdefe74556b610fed139c8373d6ae3cc0b0071473059b"}
Oct 31 19:39:56.756023 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.755872    4484 scope.go:122] "RemoveContainer" containerID="8aaf1882962d417e01bc9eb10b9956b6ca5f99d9c5156515f80044d97c7f0203"
Oct 31 19:39:56.756023 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.755835    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"
Oct 31 19:39:56.931534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.931447    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"]
Oct 31 19:39:56.943186 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:56.943155    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a8587d-998f9c9d7-mcgqg"]
Oct 31 19:39:57.993103 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:57.993068    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65891533-3ba8-44da-b413-d6936580f382" path="/var/lib/kubelet/pods/65891533-3ba8-44da-b413-d6936580f382/volumes"
Oct 31 19:39:57.993675 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:39:57.993659    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b43632-5dbb-4491-b1b8-48c1383f7266" path="/var/lib/kubelet/pods/73b43632-5dbb-4491-b1b8-48c1383f7266/volumes"
Oct 31 19:40:01.186580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:01.186532    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:01.461284 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:01.461205    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:03.657884 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:03.657848    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:03.954729 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:03.954644    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:04.451357 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:04.451316    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"]
Oct 31 19:40:04.693791 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:04.693755    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:04.950918 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:04.950875    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:05.672491 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:05.672441    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:06.903223 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:06.903182    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:07.415487 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:07.415432    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:07.691071 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:07.690989    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:09.451719 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:09.451682    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc" podUID="7f6a2a64-1451-4acc-8caf-17f9b16c30e8" containerName="payload-pre-processing" containerID="cri-o://cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444" gracePeriod=25
Oct 31 19:40:10.302713 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:10.302676    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:10.316234 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:10.316201    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:40:11.473956 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:11.473924    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:12.624809 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:12.624773    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:14.968129 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:14.968094    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc" podUID="7f6a2a64-1451-4acc-8caf-17f9b16c30e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:40:17.424388 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:17.424346    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:21.493665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.493632    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"]
Oct 31 19:40:21.494123 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.493923    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p" podUID="7d25c45a-e75d-4299-a206-c3875e851f43" containerName="maas-api" containerID="cri-o://a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e" gracePeriod=30
Oct 31 19:40:21.580622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.580588    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:21.767267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.767243    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:40:21.904555 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.904520    4484 generic.go:356] "Generic (PLEG): container finished" podID="7d25c45a-e75d-4299-a206-c3875e851f43" containerID="a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e" exitCode=0
Oct 31 19:40:21.904754 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.904582    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"
Oct 31 19:40:21.904754 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.904609    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p" event={"ID":"7d25c45a-e75d-4299-a206-c3875e851f43","Type":"ContainerDied","Data":"a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e"}
Oct 31 19:40:21.904754 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.904681    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p" event={"ID":"7d25c45a-e75d-4299-a206-c3875e851f43","Type":"ContainerDied","Data":"66f2f2f71a6dcfc72150464e35cae22f53be9f54aa1ed3740b59d332f14cdb6c"}
Oct 31 19:40:21.904754 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.904716    4484 scope.go:122] "RemoveContainer" containerID="a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e"
Oct 31 19:40:21.913246 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.913225    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d25c45a-e75d-4299-a206-c3875e851f43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d25c45a-e75d-4299-a206-c3875e851f43-openshift-service-ca\") pod \"7d25c45a-e75d-4299-a206-c3875e851f43\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") "
Oct 31 19:40:21.913387 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.913370    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-maas-api-tls\") pod \"7d25c45a-e75d-4299-a206-c3875e851f43\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") "
Oct 31 19:40:21.913449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.913411    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-metrics-certs\") pod \"7d25c45a-e75d-4299-a206-c3875e851f43\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") "
Oct 31 19:40:21.915630 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.913529    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d25c45a-e75d-4299-a206-c3875e851f43-kube-api-access-dlwhf\" (UniqueName: \"kubernetes.io/projected/7d25c45a-e75d-4299-a206-c3875e851f43-kube-api-access-dlwhf\") pod \"7d25c45a-e75d-4299-a206-c3875e851f43\" (UID: \"7d25c45a-e75d-4299-a206-c3875e851f43\") "
Oct 31 19:40:21.915630 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.913658    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d25c45a-e75d-4299-a206-c3875e851f43-openshift-service-ca" pod "7d25c45a-e75d-4299-a206-c3875e851f43" (UID: "7d25c45a-e75d-4299-a206-c3875e851f43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:21.915630 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.913979    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d25c45a-e75d-4299-a206-c3875e851f43-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:21.915878 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.915863    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-maas-api-tls" pod "7d25c45a-e75d-4299-a206-c3875e851f43" (UID: "7d25c45a-e75d-4299-a206-c3875e851f43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:21.916104 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.916076    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-metrics-certs" pod "7d25c45a-e75d-4299-a206-c3875e851f43" (UID: "7d25c45a-e75d-4299-a206-c3875e851f43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:21.916224 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.916191    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d25c45a-e75d-4299-a206-c3875e851f43-kube-api-access-dlwhf" pod "7d25c45a-e75d-4299-a206-c3875e851f43" (UID: "7d25c45a-e75d-4299-a206-c3875e851f43"). InnerVolumeSpecName "kube-api-access-dlwhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:21.917290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.917268    4484 scope.go:122] "RemoveContainer" containerID="a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e"
Oct 31 19:40:21.917701 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:40:21.917671    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e\": container with ID starting with a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e not found: ID does not exist" containerID="a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e"
Oct 31 19:40:21.917799 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:21.917710    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e"} err="failed to get container status \"a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e\": rpc error: code = NotFound desc = could not find container \"a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e\": container with ID starting with a92c81c950cfb1d9d2fbae4ccf02640c66f3517e4d2282bcfba33501cd43952e not found: ID does not exist"
Oct 31 19:40:22.015589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.015553    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:22.015589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.015590    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d25c45a-e75d-4299-a206-c3875e851f43-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:22.015806 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.015607    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlwhf\" (UniqueName: \"kubernetes.io/projected/7d25c45a-e75d-4299-a206-c3875e851f43-kube-api-access-dlwhf\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:22.321290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.321197    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"]
Oct 31 19:40:22.325045 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.325013    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d22e36-7664c56545-52p4p"]
Oct 31 19:40:22.621807 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.621726    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"]
Oct 31 19:40:22.622234 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.621951    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq" podUID="24073104-4724-49d2-8056-64b9ffc3870b" containerName="maas-api" containerID="cri-o://84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac" gracePeriod=30
Oct 31 19:40:22.709455 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.709418    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:22.887687 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.887631    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 19:40:22.912192 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.912159    4484 generic.go:356] "Generic (PLEG): container finished" podID="24073104-4724-49d2-8056-64b9ffc3870b" containerID="84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac" exitCode=0
Oct 31 19:40:22.912337 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.912206    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq" event={"ID":"24073104-4724-49d2-8056-64b9ffc3870b","Type":"ContainerDied","Data":"84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac"}
Oct 31 19:40:22.912337 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.912210    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"
Oct 31 19:40:22.912337 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.912252    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq" event={"ID":"24073104-4724-49d2-8056-64b9ffc3870b","Type":"ContainerDied","Data":"6aad574058cc8f77789e1811797f14b012205f53eea64defe28b3e4cafcc5d93"}
Oct 31 19:40:22.912337 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.912275    4484 scope.go:122] "RemoveContainer" containerID="84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac"
Oct 31 19:40:22.921343 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.921327    4484 scope.go:122] "RemoveContainer" containerID="84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac"
Oct 31 19:40:22.921670 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:40:22.921651    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac\": container with ID starting with 84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac not found: ID does not exist" containerID="84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac"
Oct 31 19:40:22.921724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:22.921679    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac"} err="failed to get container status \"84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac\": rpc error: code = NotFound desc = could not find container \"84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac\": container with ID starting with 84ae54d5c2e414cae4cec6712072aa4cbc463ad45534f482a39163980c241cac not found: ID does not exist"
Oct 31 19:40:23.027212 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.027178    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-maas-api-tls\") pod \"24073104-4724-49d2-8056-64b9ffc3870b\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") "
Oct 31 19:40:23.027422 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.027286    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24073104-4724-49d2-8056-64b9ffc3870b-kube-api-access-2mlcn\" (UniqueName: \"kubernetes.io/projected/24073104-4724-49d2-8056-64b9ffc3870b-kube-api-access-2mlcn\") pod \"24073104-4724-49d2-8056-64b9ffc3870b\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") "
Oct 31 19:40:23.027422 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.027359    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-metrics-certs\") pod \"24073104-4724-49d2-8056-64b9ffc3870b\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") "
Oct 31 19:40:23.027422 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.027387    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24073104-4724-49d2-8056-64b9ffc3870b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24073104-4724-49d2-8056-64b9ffc3870b-openshift-service-ca\") pod \"24073104-4724-49d2-8056-64b9ffc3870b\" (UID: \"24073104-4724-49d2-8056-64b9ffc3870b\") "
Oct 31 19:40:23.028165 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.028137    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24073104-4724-49d2-8056-64b9ffc3870b-openshift-service-ca" pod "24073104-4724-49d2-8056-64b9ffc3870b" (UID: "24073104-4724-49d2-8056-64b9ffc3870b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:23.030126 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.030093    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-metrics-certs" pod "24073104-4724-49d2-8056-64b9ffc3870b" (UID: "24073104-4724-49d2-8056-64b9ffc3870b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:23.030251 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.030230    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-maas-api-tls" pod "24073104-4724-49d2-8056-64b9ffc3870b" (UID: "24073104-4724-49d2-8056-64b9ffc3870b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:23.030338 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.030315    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24073104-4724-49d2-8056-64b9ffc3870b-kube-api-access-2mlcn" pod "24073104-4724-49d2-8056-64b9ffc3870b" (UID: "24073104-4724-49d2-8056-64b9ffc3870b"). InnerVolumeSpecName "kube-api-access-2mlcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:23.129304 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.129271    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:23.129304 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.129304    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mlcn\" (UniqueName: \"kubernetes.io/projected/24073104-4724-49d2-8056-64b9ffc3870b-kube-api-access-2mlcn\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:23.129610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.129324    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24073104-4724-49d2-8056-64b9ffc3870b-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:23.129610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.129333    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24073104-4724-49d2-8056-64b9ffc3870b-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:23.339034 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.339003    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"]
Oct 31 19:40:23.342677 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.342650    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a8587d-579bcfb695-qnzcq"]
Oct 31 19:40:23.993401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.993372    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24073104-4724-49d2-8056-64b9ffc3870b" path="/var/lib/kubelet/pods/24073104-4724-49d2-8056-64b9ffc3870b/volumes"
Oct 31 19:40:23.993875 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:23.993853    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d25c45a-e75d-4299-a206-c3875e851f43" path="/var/lib/kubelet/pods/7d25c45a-e75d-4299-a206-c3875e851f43/volumes"
Oct 31 19:40:24.524228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:24.524196    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:24.967615 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:24.967532    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc" podUID="7f6a2a64-1451-4acc-8caf-17f9b16c30e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:40:25.469956 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:40:25.469921    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:40:25.650927 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:25.650884    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:26.105963 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:26.105928    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:26.466353 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:26.466249    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:27.265458 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:27.265423    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:27.805088 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:27.805048    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:28.266768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:28.266733    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:28.658365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:28.658273    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:29.164909 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.164846    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:29.201193 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.201128    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"]
Oct 31 19:40:29.201410 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.201374    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq" podUID="c3ee3c68-0043-49ac-be93-fccb0cb0f932" containerName="istio-proxy" containerID="cri-o://e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0" gracePeriod=30
Oct 31 19:40:29.722616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.722550    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"
Oct 31 19:40:29.736040 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.736014    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:29.799502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.799444    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-plugins-config-volume\") pod \"7f6a2a64-1451-4acc-8caf-17f9b16c30e8\" (UID: \"7f6a2a64-1451-4acc-8caf-17f9b16c30e8\") "
Oct 31 19:40:29.799685 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.799573    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-kube-api-access-6xw2x\" (UniqueName: \"kubernetes.io/projected/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-kube-api-access-6xw2x\") pod \"7f6a2a64-1451-4acc-8caf-17f9b16c30e8\" (UID: \"7f6a2a64-1451-4acc-8caf-17f9b16c30e8\") "
Oct 31 19:40:29.799855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.799833    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-plugins-config-volume" pod "7f6a2a64-1451-4acc-8caf-17f9b16c30e8" (UID: "7f6a2a64-1451-4acc-8caf-17f9b16c30e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:29.800017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.799989    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:29.801977 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.801955    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-kube-api-access-6xw2x" pod "7f6a2a64-1451-4acc-8caf-17f9b16c30e8" (UID: "7f6a2a64-1451-4acc-8caf-17f9b16c30e8"). InnerVolumeSpecName "kube-api-access-6xw2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:29.901226 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.901195    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xw2x\" (UniqueName: \"kubernetes.io/projected/7f6a2a64-1451-4acc-8caf-17f9b16c30e8-kube-api-access-6xw2x\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:29.954616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.954586    4484 generic.go:356] "Generic (PLEG): container finished" podID="7f6a2a64-1451-4acc-8caf-17f9b16c30e8" containerID="cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444" exitCode=0
Oct 31 19:40:29.954781 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.954634    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"
Oct 31 19:40:29.954781 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.954643    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc" event={"ID":"7f6a2a64-1451-4acc-8caf-17f9b16c30e8","Type":"ContainerDied","Data":"cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444"}
Oct 31 19:40:29.954781 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.954684    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc" event={"ID":"7f6a2a64-1451-4acc-8caf-17f9b16c30e8","Type":"ContainerDied","Data":"27acfc2093b903ff6e4c8aee6560150034626c4fcde69e9aa6c6f2c99a658e9a"}
Oct 31 19:40:29.954781 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.954706    4484 scope.go:122] "RemoveContainer" containerID="cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444"
Oct 31 19:40:29.964630 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.964613    4484 scope.go:122] "RemoveContainer" containerID="cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444"
Oct 31 19:40:29.964900 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:40:29.964878    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444\": container with ID starting with cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444 not found: ID does not exist" containerID="cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444"
Oct 31 19:40:29.964943 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:29.964910    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444"} err="failed to get container status \"cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444\": rpc error: code = NotFound desc = could not find container \"cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444\": container with ID starting with cfb5d7dff726e606fb6981b72b0d5510c603bb5d1b2adcd9af2127a5679ad444 not found: ID does not exist"
Oct 31 19:40:30.039859 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:30.039828    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:30.122881 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:30.122844    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"]
Oct 31 19:40:30.125026 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:30.125004    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d1ff32-b4ddbdd67-xbgtc"]
Oct 31 19:40:32.000058 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:32.000020    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f6a2a64-1451-4acc-8caf-17f9b16c30e8" path="/var/lib/kubelet/pods/7f6a2a64-1451-4acc-8caf-17f9b16c30e8/volumes"
Oct 31 19:40:33.242605 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:33.242573    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"]
Oct 31 19:40:33.294158 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:33.294094    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"]
Oct 31 19:40:33.297454 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:33.297424    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"]
Oct 31 19:40:33.299868 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:33.299843    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"]
Oct 31 19:40:33.419624 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:33.419588    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:33.844290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:33.844258    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:34.466677 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.466650    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 19:40:34.549492 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.549442    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-socket\") pod \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") "
Oct 31 19:40:34.549691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.549545    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-kube-api-access-dbfcz\" (UniqueName: \"kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-kube-api-access-dbfcz\") pod \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") "
Oct 31 19:40:34.549691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.549606    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istiod-ca-cert\") pod \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") "
Oct 31 19:40:34.549691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.549635    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-envoy\") pod \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") "
Oct 31 19:40:34.549861 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.549715    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-certs\") pod \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") "
Oct 31 19:40:34.549861 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.549743    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-token\" (UniqueName: \"kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-token\") pod \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") "
Oct 31 19:40:34.549861 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.549716    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-socket" pod "c3ee3c68-0043-49ac-be93-fccb0cb0f932" (UID: "c3ee3c68-0043-49ac-be93-fccb0cb0f932"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:34.549861 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.549784    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-podinfo\") pod \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") "
Oct 31 19:40:34.549861 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.549853    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-credential-socket\") pod \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") "
Oct 31 19:40:34.550122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.549892    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-data\") pod \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\" (UID: \"c3ee3c68-0043-49ac-be93-fccb0cb0f932\") "
Oct 31 19:40:34.550122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.550013    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-certs" pod "c3ee3c68-0043-49ac-be93-fccb0cb0f932" (UID: "c3ee3c68-0043-49ac-be93-fccb0cb0f932"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:34.550231 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.550147    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istiod-ca-cert" pod "c3ee3c68-0043-49ac-be93-fccb0cb0f932" (UID: "c3ee3c68-0043-49ac-be93-fccb0cb0f932"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:34.550279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.550260    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-credential-socket" pod "c3ee3c68-0043-49ac-be93-fccb0cb0f932" (UID: "c3ee3c68-0043-49ac-be93-fccb0cb0f932"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:34.550279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.550267    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-data" pod "c3ee3c68-0043-49ac-be93-fccb0cb0f932" (UID: "c3ee3c68-0043-49ac-be93-fccb0cb0f932"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:34.550388 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.550323    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.550388 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.550343    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.550388 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.550357    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.551952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.551933    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-envoy" pod "c3ee3c68-0043-49ac-be93-fccb0cb0f932" (UID: "c3ee3c68-0043-49ac-be93-fccb0cb0f932"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:34.552235 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.552217    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-token" pod "c3ee3c68-0043-49ac-be93-fccb0cb0f932" (UID: "c3ee3c68-0043-49ac-be93-fccb0cb0f932"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:34.552456 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.552439    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-podinfo" pod "c3ee3c68-0043-49ac-be93-fccb0cb0f932" (UID: "c3ee3c68-0043-49ac-be93-fccb0cb0f932"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:40:34.552535 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.552507    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-kube-api-access-dbfcz" pod "c3ee3c68-0043-49ac-be93-fccb0cb0f932" (UID: "c3ee3c68-0043-49ac-be93-fccb0cb0f932"). InnerVolumeSpecName "kube-api-access-dbfcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:34.651750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.651715    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbfcz\" (UniqueName: \"kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-kube-api-access-dbfcz\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.651750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.651743    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.651750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.651752    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.651750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.651761    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.652120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.651770    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.652120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.651778    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ee3c68-0043-49ac-be93-fccb0cb0f932-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:34.929996 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.929898    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:34.986441 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.986410    4484 generic.go:356] "Generic (PLEG): container finished" podID="c3ee3c68-0043-49ac-be93-fccb0cb0f932" containerID="e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0" exitCode=0
Oct 31 19:40:34.986662 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.986464    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"
Oct 31 19:40:34.986662 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.986502    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq" event={"ID":"c3ee3c68-0043-49ac-be93-fccb0cb0f932","Type":"ContainerDied","Data":"e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0"}
Oct 31 19:40:34.986662 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.986554    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq" event={"ID":"c3ee3c68-0043-49ac-be93-fccb0cb0f932","Type":"ContainerDied","Data":"d575ea0463b1026d9255002a41fd2714169a3f343b403a0fd68b46a3bdda8002"}
Oct 31 19:40:34.986662 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:34.986576    4484 scope.go:122] "RemoveContainer" containerID="e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0"
Oct 31 19:40:35.000541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:35.000515    4484 scope.go:122] "RemoveContainer" containerID="e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0"
Oct 31 19:40:35.000991 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:40:35.000967    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0\": container with ID starting with e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0 not found: ID does not exist" containerID="e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0"
Oct 31 19:40:35.001098 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:35.001000    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0"} err="failed to get container status \"e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0\": rpc error: code = NotFound desc = could not find container \"e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0\": container with ID starting with e3a4a3237c813cde4b40d5581f1d768f8adc7d2f77ad7aec72cd495d4b392de0 not found: ID does not exist"
Oct 31 19:40:35.157342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:35.157302    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"]
Oct 31 19:40:35.160466 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:35.160440    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a8587d-openshift-default-5cd77d84f5-sf8fq"]
Oct 31 19:40:35.993184 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:35.993155    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ee3c68-0043-49ac-be93-fccb0cb0f932" path="/var/lib/kubelet/pods/c3ee3c68-0043-49ac-be93-fccb0cb0f932/volumes"
Oct 31 19:40:38.477929 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:38.477892    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:39.636256 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:39.636221    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:40.258065 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:40.258033    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:40.859256 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:40.859222    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"]
Oct 31 19:40:40.966814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:40.966771    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"]
Oct 31 19:40:41.213196 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:41.213118    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:41.532972 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:41.532931    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:42.005484 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:42.005433    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:42.249515 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:42.246557    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:44.107105 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:44.107067    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:44.724524 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:44.724453    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:45.859526 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:45.859463    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz" podUID="1142e654-6da9-46c3-a10e-c84bdd017f78" containerName="payload-processing" containerID="cri-o://a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe" gracePeriod=25
Oct 31 19:40:45.968072 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:45.968034    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws" podUID="50dad950-53be-49e3-883b-d803ac7858df" containerName="payload-pre-processing" containerID="cri-o://bc2a0f564dec4353e855a72ef2421569b70401cda30ff554c8df100e5d991a64" gracePeriod=25
Oct 31 19:40:46.223552 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:46.223431    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:46.238820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:46.238790    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws" podUID="50dad950-53be-49e3-883b-d803ac7858df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:40:48.263122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.263059    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c" podUID="307a9ca2-cb52-4131-b750-2fb5d89e8235" containerName="main" containerID="cri-o://d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1" gracePeriod=45
Oct 31 19:40:48.315118 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.315079    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4" podUID="7fdc08a1-c720-4038-acca-b931fbec7633" containerName="main" containerID="cri-o://88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a" gracePeriod=45
Oct 31 19:40:48.318846 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.318818    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr" podUID="020ad937-bef8-41ae-9394-eab2f30eefba" containerName="main" containerID="cri-o://d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae" gracePeriod=45
Oct 31 19:40:48.322954 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.322688    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv" podUID="3b6b07cc-6154-4582-8e24-fc085949657a" containerName="main" containerID="cri-o://d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926" gracePeriod=45
Oct 31 19:40:48.605810 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.605786    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 19:40:48.632153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.632119    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 19:40:48.638969 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.638949    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 19:40:48.644442 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.644420    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 19:40:48.701665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.701629    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-dshm\") pod \"020ad937-bef8-41ae-9394-eab2f30eefba\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") "
Oct 31 19:40:48.701665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.701662    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/020ad937-bef8-41ae-9394-eab2f30eefba-tls-certs\" (UniqueName: \"kubernetes.io/secret/020ad937-bef8-41ae-9394-eab2f30eefba-tls-certs\") pod \"020ad937-bef8-41ae-9394-eab2f30eefba\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") "
Oct 31 19:40:48.701923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.701702    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fdc08a1-c720-4038-acca-b931fbec7633-tls-certs\" (UniqueName: \"kubernetes.io/secret/7fdc08a1-c720-4038-acca-b931fbec7633-tls-certs\") pod \"7fdc08a1-c720-4038-acca-b931fbec7633\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") "
Oct 31 19:40:48.701923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.701724    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020ad937-bef8-41ae-9394-eab2f30eefba-kube-api-access-7dm8k\" (UniqueName: \"kubernetes.io/projected/020ad937-bef8-41ae-9394-eab2f30eefba-kube-api-access-7dm8k\") pod \"020ad937-bef8-41ae-9394-eab2f30eefba\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") "
Oct 31 19:40:48.701923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.701745    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-home\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-home\") pod \"7fdc08a1-c720-4038-acca-b931fbec7633\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") "
Oct 31 19:40:48.701923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.701765    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fdc08a1-c720-4038-acca-b931fbec7633-kube-api-access-vwt25\" (UniqueName: \"kubernetes.io/projected/7fdc08a1-c720-4038-acca-b931fbec7633-kube-api-access-vwt25\") pod \"7fdc08a1-c720-4038-acca-b931fbec7633\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") "
Oct 31 19:40:48.701923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.701851    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-home\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-home\") pod \"020ad937-bef8-41ae-9394-eab2f30eefba\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") "
Oct 31 19:40:48.701923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.701907    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-model-cache\") pod \"020ad937-bef8-41ae-9394-eab2f30eefba\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") "
Oct 31 19:40:48.702286 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.701946    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-dshm\") pod \"7fdc08a1-c720-4038-acca-b931fbec7633\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") "
Oct 31 19:40:48.702286 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.702005    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-tmp-dir\") pod \"7fdc08a1-c720-4038-acca-b931fbec7633\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") "
Oct 31 19:40:48.702286 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.702064    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-model-cache\") pod \"7fdc08a1-c720-4038-acca-b931fbec7633\" (UID: \"7fdc08a1-c720-4038-acca-b931fbec7633\") "
Oct 31 19:40:48.702286 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.702114    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-tmp-dir\") pod \"020ad937-bef8-41ae-9394-eab2f30eefba\" (UID: \"020ad937-bef8-41ae-9394-eab2f30eefba\") "
Oct 31 19:40:48.702286 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.702166    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-home" pod "7fdc08a1-c720-4038-acca-b931fbec7633" (UID: "7fdc08a1-c720-4038-acca-b931fbec7633"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.702579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.702467    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-tmp-dir" pod "7fdc08a1-c720-4038-acca-b931fbec7633" (UID: "7fdc08a1-c720-4038-acca-b931fbec7633"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.702675 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.702638    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.702752 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.702691    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.702811 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.702732    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-tmp-dir" pod "020ad937-bef8-41ae-9394-eab2f30eefba" (UID: "020ad937-bef8-41ae-9394-eab2f30eefba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.702874 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.702813    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-model-cache" pod "7fdc08a1-c720-4038-acca-b931fbec7633" (UID: "7fdc08a1-c720-4038-acca-b931fbec7633"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.702939 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.702906    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-home" pod "020ad937-bef8-41ae-9394-eab2f30eefba" (UID: "020ad937-bef8-41ae-9394-eab2f30eefba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.702984 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.702957    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-model-cache" pod "020ad937-bef8-41ae-9394-eab2f30eefba" (UID: "020ad937-bef8-41ae-9394-eab2f30eefba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.704958 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.704938    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-dshm" pod "020ad937-bef8-41ae-9394-eab2f30eefba" (UID: "020ad937-bef8-41ae-9394-eab2f30eefba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.705064 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.704979    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020ad937-bef8-41ae-9394-eab2f30eefba-kube-api-access-7dm8k" pod "020ad937-bef8-41ae-9394-eab2f30eefba" (UID: "020ad937-bef8-41ae-9394-eab2f30eefba"). InnerVolumeSpecName "kube-api-access-7dm8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:48.705064 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.705055    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fdc08a1-c720-4038-acca-b931fbec7633-tls-certs" pod "7fdc08a1-c720-4038-acca-b931fbec7633" (UID: "7fdc08a1-c720-4038-acca-b931fbec7633"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:48.705171 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.705052    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020ad937-bef8-41ae-9394-eab2f30eefba-tls-certs" pod "020ad937-bef8-41ae-9394-eab2f30eefba" (UID: "020ad937-bef8-41ae-9394-eab2f30eefba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:48.705171 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.705072    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-dshm" pod "7fdc08a1-c720-4038-acca-b931fbec7633" (UID: "7fdc08a1-c720-4038-acca-b931fbec7633"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.705171 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.705106    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdc08a1-c720-4038-acca-b931fbec7633-kube-api-access-vwt25" pod "7fdc08a1-c720-4038-acca-b931fbec7633" (UID: "7fdc08a1-c720-4038-acca-b931fbec7633"). InnerVolumeSpecName "kube-api-access-vwt25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:48.803312 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.803228    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b6b07cc-6154-4582-8e24-fc085949657a-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6b07cc-6154-4582-8e24-fc085949657a-tls-certs\") pod \"3b6b07cc-6154-4582-8e24-fc085949657a\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") "
Oct 31 19:40:48.803312 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.803308    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-home\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-home\") pod \"307a9ca2-cb52-4131-b750-2fb5d89e8235\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") "
Oct 31 19:40:48.803579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.803381    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6b07cc-6154-4582-8e24-fc085949657a-kube-api-access-dwnjf\" (UniqueName: \"kubernetes.io/projected/3b6b07cc-6154-4582-8e24-fc085949657a-kube-api-access-dwnjf\") pod \"3b6b07cc-6154-4582-8e24-fc085949657a\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") "
Oct 31 19:40:48.803579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.803407    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-model-cache\") pod \"307a9ca2-cb52-4131-b750-2fb5d89e8235\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") "
Oct 31 19:40:48.803579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.803450    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-tmp-dir\") pod \"307a9ca2-cb52-4131-b750-2fb5d89e8235\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") "
Oct 31 19:40:48.803579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.803501    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-dshm\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-dshm\") pod \"307a9ca2-cb52-4131-b750-2fb5d89e8235\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") "
Oct 31 19:40:48.803579 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.803558    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307a9ca2-cb52-4131-b750-2fb5d89e8235-kube-api-access-cp9dn\" (UniqueName: \"kubernetes.io/projected/307a9ca2-cb52-4131-b750-2fb5d89e8235-kube-api-access-cp9dn\") pod \"307a9ca2-cb52-4131-b750-2fb5d89e8235\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") "
Oct 31 19:40:48.803857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.803573    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-home" pod "307a9ca2-cb52-4131-b750-2fb5d89e8235" (UID: "307a9ca2-cb52-4131-b750-2fb5d89e8235"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.803857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.803707    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-model-cache" pod "307a9ca2-cb52-4131-b750-2fb5d89e8235" (UID: "307a9ca2-cb52-4131-b750-2fb5d89e8235"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.803857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.803728    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-tmp-dir" pod "307a9ca2-cb52-4131-b750-2fb5d89e8235" (UID: "307a9ca2-cb52-4131-b750-2fb5d89e8235"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.804017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.803956    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-tmp-dir\") pod \"3b6b07cc-6154-4582-8e24-fc085949657a\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") "
Oct 31 19:40:48.804017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804002    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-model-cache\") pod \"3b6b07cc-6154-4582-8e24-fc085949657a\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") "
Oct 31 19:40:48.804130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804037    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-home\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-home\") pod \"3b6b07cc-6154-4582-8e24-fc085949657a\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") "
Oct 31 19:40:48.804130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804075    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-dshm\") pod \"3b6b07cc-6154-4582-8e24-fc085949657a\" (UID: \"3b6b07cc-6154-4582-8e24-fc085949657a\") "
Oct 31 19:40:48.804130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804102    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/307a9ca2-cb52-4131-b750-2fb5d89e8235-tls-certs\" (UniqueName: \"kubernetes.io/secret/307a9ca2-cb52-4131-b750-2fb5d89e8235-tls-certs\") pod \"307a9ca2-cb52-4131-b750-2fb5d89e8235\" (UID: \"307a9ca2-cb52-4131-b750-2fb5d89e8235\") "
Oct 31 19:40:48.804274 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804255    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-tmp-dir" pod "3b6b07cc-6154-4582-8e24-fc085949657a" (UID: "3b6b07cc-6154-4582-8e24-fc085949657a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.804520 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804499    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-model-cache" pod "3b6b07cc-6154-4582-8e24-fc085949657a" (UID: "3b6b07cc-6154-4582-8e24-fc085949657a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804602    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804624    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804640    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804654    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fdc08a1-c720-4038-acca-b931fbec7633-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804666    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804664    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-home" pod "3b6b07cc-6154-4582-8e24-fc085949657a" (UID: "3b6b07cc-6154-4582-8e24-fc085949657a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804679    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804691    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804703    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804715    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020ad937-bef8-41ae-9394-eab2f30eefba-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804727    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020ad937-bef8-41ae-9394-eab2f30eefba-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804738    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fdc08a1-c720-4038-acca-b931fbec7633-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804751    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804766    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804779    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dm8k\" (UniqueName: \"kubernetes.io/projected/020ad937-bef8-41ae-9394-eab2f30eefba-kube-api-access-7dm8k\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.804819 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.804794    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwt25\" (UniqueName: \"kubernetes.io/projected/7fdc08a1-c720-4038-acca-b931fbec7633-kube-api-access-vwt25\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.805921 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.805889    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6b07cc-6154-4582-8e24-fc085949657a-tls-certs" pod "3b6b07cc-6154-4582-8e24-fc085949657a" (UID: "3b6b07cc-6154-4582-8e24-fc085949657a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:48.806201 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.806156    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-dshm" pod "307a9ca2-cb52-4131-b750-2fb5d89e8235" (UID: "307a9ca2-cb52-4131-b750-2fb5d89e8235"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.806282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.806246    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307a9ca2-cb52-4131-b750-2fb5d89e8235-kube-api-access-cp9dn" pod "307a9ca2-cb52-4131-b750-2fb5d89e8235" (UID: "307a9ca2-cb52-4131-b750-2fb5d89e8235"). InnerVolumeSpecName "kube-api-access-cp9dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:48.806282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.806271    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6b07cc-6154-4582-8e24-fc085949657a-kube-api-access-dwnjf" pod "3b6b07cc-6154-4582-8e24-fc085949657a" (UID: "3b6b07cc-6154-4582-8e24-fc085949657a"). InnerVolumeSpecName "kube-api-access-dwnjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:48.806604 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.806587    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307a9ca2-cb52-4131-b750-2fb5d89e8235-tls-certs" pod "307a9ca2-cb52-4131-b750-2fb5d89e8235" (UID: "307a9ca2-cb52-4131-b750-2fb5d89e8235"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:48.806670 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.806618    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-dshm" pod "3b6b07cc-6154-4582-8e24-fc085949657a" (UID: "3b6b07cc-6154-4582-8e24-fc085949657a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:48.906042 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.906016    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.906042 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.906040    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6b07cc-6154-4582-8e24-fc085949657a-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.906042 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.906051    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/307a9ca2-cb52-4131-b750-2fb5d89e8235-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.906279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.906059    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6b07cc-6154-4582-8e24-fc085949657a-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.906279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.906069    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwnjf\" (UniqueName: \"kubernetes.io/projected/3b6b07cc-6154-4582-8e24-fc085949657a-kube-api-access-dwnjf\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.906279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.906078    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/307a9ca2-cb52-4131-b750-2fb5d89e8235-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:48.906279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:48.906086    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp9dn\" (UniqueName: \"kubernetes.io/projected/307a9ca2-cb52-4131-b750-2fb5d89e8235-kube-api-access-cp9dn\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:40:49.069426 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.069342    4484 generic.go:356] "Generic (PLEG): container finished" podID="020ad937-bef8-41ae-9394-eab2f30eefba" containerID="d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae" exitCode=0
Oct 31 19:40:49.069426 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.069393    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"
Oct 31 19:40:49.069676 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.069431    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr" event={"ID":"020ad937-bef8-41ae-9394-eab2f30eefba","Type":"ContainerDied","Data":"d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae"}
Oct 31 19:40:49.069676 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.069493    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr" event={"ID":"020ad937-bef8-41ae-9394-eab2f30eefba","Type":"ContainerDied","Data":"88ca0d352d1b1cb2af1a07fff749cb85580df2332084dfe3fb6f8bfcf916c898"}
Oct 31 19:40:49.069676 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.069516    4484 scope.go:122] "RemoveContainer" containerID="d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae"
Oct 31 19:40:49.070883 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.070854    4484 generic.go:356] "Generic (PLEG): container finished" podID="7fdc08a1-c720-4038-acca-b931fbec7633" containerID="88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a" exitCode=0
Oct 31 19:40:49.071015 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.070910    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"
Oct 31 19:40:49.071015 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.070922    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4" event={"ID":"7fdc08a1-c720-4038-acca-b931fbec7633","Type":"ContainerDied","Data":"88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a"}
Oct 31 19:40:49.071015 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.070961    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4" event={"ID":"7fdc08a1-c720-4038-acca-b931fbec7633","Type":"ContainerDied","Data":"b2264dded3c0f822be5db183dcc8943afc982a98ad53dcb9cb30f84c08d6a80e"}
Oct 31 19:40:49.072509 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.072452    4484 generic.go:356] "Generic (PLEG): container finished" podID="3b6b07cc-6154-4582-8e24-fc085949657a" containerID="d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926" exitCode=0
Oct 31 19:40:49.072649 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.072528    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv" event={"ID":"3b6b07cc-6154-4582-8e24-fc085949657a","Type":"ContainerDied","Data":"d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926"}
Oct 31 19:40:49.072649 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.072544    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"
Oct 31 19:40:49.072649 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.072556    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv" event={"ID":"3b6b07cc-6154-4582-8e24-fc085949657a","Type":"ContainerDied","Data":"b4a8852bef2639c6b1b814673575b3e7c864b3c2ee5dbed393aac518be20d583"}
Oct 31 19:40:49.074212 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.074189    4484 generic.go:356] "Generic (PLEG): container finished" podID="307a9ca2-cb52-4131-b750-2fb5d89e8235" containerID="d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1" exitCode=0
Oct 31 19:40:49.078888 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.074408    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"
Oct 31 19:40:49.078888 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.074424    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c" event={"ID":"307a9ca2-cb52-4131-b750-2fb5d89e8235","Type":"ContainerDied","Data":"d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1"}
Oct 31 19:40:49.078888 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.074456    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c" event={"ID":"307a9ca2-cb52-4131-b750-2fb5d89e8235","Type":"ContainerDied","Data":"2a4649f204f1aebefe54fbc3ee9d2fdb1a303eabc196077317756378f83434ee"}
Oct 31 19:40:49.081883 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.081814    4484 scope.go:122] "RemoveContainer" containerID="d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae"
Oct 31 19:40:49.082178 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:40:49.082141    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae\": container with ID starting with d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae not found: ID does not exist" containerID="d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae"
Oct 31 19:40:49.082248 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.082171    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae"} err="failed to get container status \"d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae\": rpc error: code = NotFound desc = could not find container \"d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae\": container with ID starting with d640b964df185fc817286fe28a6c5e1571a2e5c57b26e3cc40fc94ad15a3c9ae not found: ID does not exist"
Oct 31 19:40:49.082248 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.082193    4484 scope.go:122] "RemoveContainer" containerID="88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a"
Oct 31 19:40:49.096467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.096442    4484 scope.go:122] "RemoveContainer" containerID="88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a"
Oct 31 19:40:49.096880 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:40:49.096859    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a\": container with ID starting with 88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a not found: ID does not exist" containerID="88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a"
Oct 31 19:40:49.096953 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.096887    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a"} err="failed to get container status \"88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a\": rpc error: code = NotFound desc = could not find container \"88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a\": container with ID starting with 88476913dbe770e70baf26aeddcd97f8f8bb1e54197091a142e62d9be1222d9a not found: ID does not exist"
Oct 31 19:40:49.096953 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.096904    4484 scope.go:122] "RemoveContainer" containerID="d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926"
Oct 31 19:40:49.105638 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.105612    4484 scope.go:122] "RemoveContainer" containerID="d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926"
Oct 31 19:40:49.105935 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:40:49.105917    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926\": container with ID starting with d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926 not found: ID does not exist" containerID="d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926"
Oct 31 19:40:49.106042 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.105939    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926"} err="failed to get container status \"d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926\": rpc error: code = NotFound desc = could not find container \"d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926\": container with ID starting with d6bc55c389c60153c050a20ceaf0dc556bcf034c54153673e819d17432469926 not found: ID does not exist"
Oct 31 19:40:49.106042 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.105956    4484 scope.go:122] "RemoveContainer" containerID="d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1"
Oct 31 19:40:49.114543 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.114524    4484 scope.go:122] "RemoveContainer" containerID="d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1"
Oct 31 19:40:49.114852 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:40:49.114834    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1\": container with ID starting with d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1 not found: ID does not exist" containerID="d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1"
Oct 31 19:40:49.115065 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.114858    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1"} err="failed to get container status \"d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1\": rpc error: code = NotFound desc = could not find container \"d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1\": container with ID starting with d672f99749058b20694e66bfec37388a47ad64056391eceafd1af55af0e049b1 not found: ID does not exist"
Oct 31 19:40:49.405401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.405325    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"]
Oct 31 19:40:49.409020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.408996    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/simulator-w3-3e4a2c-kserve-86d47f776b-qblsv"]
Oct 31 19:40:49.524573 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.524536    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"]
Oct 31 19:40:49.529080 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.529054    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/distinct-2-w3-3e4a2c-kserve-849f794f69-c2b6c"]
Oct 31 19:40:49.545918 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.545887    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:49.597146 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.597108    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4" podUID="7fdc08a1-c720-4038-acca-b931fbec7633" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:40:49.641670 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.641642    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"]
Oct 31 19:40:49.644924 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.644899    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/distinct-w3-3e4a2c-kserve-55f65fc498-4jhf4"]
Oct 31 19:40:49.756949 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.756919    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"]
Oct 31 19:40:49.760370 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.760349    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3e4a2c/premium-w3-3e4a2c-kserve-6b7f89577f-dk2pr"]
Oct 31 19:40:49.993053 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.993016    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="020ad937-bef8-41ae-9394-eab2f30eefba" path="/var/lib/kubelet/pods/020ad937-bef8-41ae-9394-eab2f30eefba/volumes"
Oct 31 19:40:49.993443 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.993429    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307a9ca2-cb52-4131-b750-2fb5d89e8235" path="/var/lib/kubelet/pods/307a9ca2-cb52-4131-b750-2fb5d89e8235/volumes"
Oct 31 19:40:49.993836 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.993822    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6b07cc-6154-4582-8e24-fc085949657a" path="/var/lib/kubelet/pods/3b6b07cc-6154-4582-8e24-fc085949657a/volumes"
Oct 31 19:40:49.994577 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:49.994559    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fdc08a1-c720-4038-acca-b931fbec7633" path="/var/lib/kubelet/pods/7fdc08a1-c720-4038-acca-b931fbec7633/volumes"
Oct 31 19:40:50.113677 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:50.113595    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:50.421029 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:50.420939    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:52.334194 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:52.334161    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:54.222449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:54.222416    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz" podUID="1142e654-6da9-46c3-a10e-c84bdd017f78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:40:54.338196 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:54.338159    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:54.908630 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:54.908595    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:40:54.915912 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:54.915890    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:40:55.467811 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:40:55.467769    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:40:56.239254 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:56.239220    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws" podUID="50dad950-53be-49e3-883b-d803ac7858df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:40:59.019583 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:40:59.019550    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:41:00.160965 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:00.160933    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:01.417755 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:01.417712    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:01.683029 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:01.682952    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:02.992095 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:02.992057    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:03.510925 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:03.510887    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:04.223241 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:04.223202    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz" podUID="1142e654-6da9-46c3-a10e-c84bdd017f78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:41:04.338130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:04.338103    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:05.505647 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:05.505602    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:06.117185 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.117123    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"
Oct 31 19:41:06.176444 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.175683    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1142e654-6da9-46c3-a10e-c84bdd017f78-kube-api-access-9q54f\" (UniqueName: \"kubernetes.io/projected/1142e654-6da9-46c3-a10e-c84bdd017f78-kube-api-access-9q54f\") pod \"1142e654-6da9-46c3-a10e-c84bdd017f78\" (UID: \"1142e654-6da9-46c3-a10e-c84bdd017f78\") "
Oct 31 19:41:06.176444 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.175772    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1142e654-6da9-46c3-a10e-c84bdd017f78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1142e654-6da9-46c3-a10e-c84bdd017f78-plugins-config-volume\") pod \"1142e654-6da9-46c3-a10e-c84bdd017f78\" (UID: \"1142e654-6da9-46c3-a10e-c84bdd017f78\") "
Oct 31 19:41:06.176444 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.176406    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1142e654-6da9-46c3-a10e-c84bdd017f78-plugins-config-volume" pod "1142e654-6da9-46c3-a10e-c84bdd017f78" (UID: "1142e654-6da9-46c3-a10e-c84bdd017f78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:06.178800 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.178773    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1142e654-6da9-46c3-a10e-c84bdd017f78-kube-api-access-9q54f" pod "1142e654-6da9-46c3-a10e-c84bdd017f78" (UID: "1142e654-6da9-46c3-a10e-c84bdd017f78"). InnerVolumeSpecName "kube-api-access-9q54f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:06.193419 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.193390    4484 generic.go:356] "Generic (PLEG): container finished" podID="50dad950-53be-49e3-883b-d803ac7858df" containerID="bc2a0f564dec4353e855a72ef2421569b70401cda30ff554c8df100e5d991a64" exitCode=0
Oct 31 19:41:06.193586 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.193457    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws" event={"ID":"50dad950-53be-49e3-883b-d803ac7858df","Type":"ContainerDied","Data":"bc2a0f564dec4353e855a72ef2421569b70401cda30ff554c8df100e5d991a64"}
Oct 31 19:41:06.194916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.194893    4484 generic.go:356] "Generic (PLEG): container finished" podID="1142e654-6da9-46c3-a10e-c84bdd017f78" containerID="a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe" exitCode=0
Oct 31 19:41:06.195049 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.194939    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"
Oct 31 19:41:06.195049 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.194977    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz" event={"ID":"1142e654-6da9-46c3-a10e-c84bdd017f78","Type":"ContainerDied","Data":"a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe"}
Oct 31 19:41:06.195049 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.195022    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz" event={"ID":"1142e654-6da9-46c3-a10e-c84bdd017f78","Type":"ContainerDied","Data":"9f7930501658e80466780b1e8bb170cbb7ecb237caa634233c3db2afa960dc9b"}
Oct 31 19:41:06.195218 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.195049    4484 scope.go:122] "RemoveContainer" containerID="a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe"
Oct 31 19:41:06.207682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.206115    4484 scope.go:122] "RemoveContainer" containerID="a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe"
Oct 31 19:41:06.207682 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:41:06.206402    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe\": container with ID starting with a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe not found: ID does not exist" containerID="a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe"
Oct 31 19:41:06.207682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.206429    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe"} err="failed to get container status \"a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe\": rpc error: code = NotFound desc = could not find container \"a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe\": container with ID starting with a86047f31b8fa48b9f224d6f65e09b242fc749ae64cabd992058b4c0659c6ebe not found: ID does not exist"
Oct 31 19:41:06.227139 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.227119    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"
Oct 31 19:41:06.276906 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.276868    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50dad950-53be-49e3-883b-d803ac7858df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50dad950-53be-49e3-883b-d803ac7858df-plugins-config-volume\") pod \"50dad950-53be-49e3-883b-d803ac7858df\" (UID: \"50dad950-53be-49e3-883b-d803ac7858df\") "
Oct 31 19:41:06.277071 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.276998    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50dad950-53be-49e3-883b-d803ac7858df-kube-api-access-82zsx\" (UniqueName: \"kubernetes.io/projected/50dad950-53be-49e3-883b-d803ac7858df-kube-api-access-82zsx\") pod \"50dad950-53be-49e3-883b-d803ac7858df\" (UID: \"50dad950-53be-49e3-883b-d803ac7858df\") "
Oct 31 19:41:06.277200 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.277189    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q54f\" (UniqueName: \"kubernetes.io/projected/1142e654-6da9-46c3-a10e-c84bdd017f78-kube-api-access-9q54f\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:06.277247 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.277203    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1142e654-6da9-46c3-a10e-c84bdd017f78-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:06.277247 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.277216    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50dad950-53be-49e3-883b-d803ac7858df-plugins-config-volume" pod "50dad950-53be-49e3-883b-d803ac7858df" (UID: "50dad950-53be-49e3-883b-d803ac7858df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:06.279249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.279230    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50dad950-53be-49e3-883b-d803ac7858df-kube-api-access-82zsx" pod "50dad950-53be-49e3-883b-d803ac7858df" (UID: "50dad950-53be-49e3-883b-d803ac7858df"). InnerVolumeSpecName "kube-api-access-82zsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:06.377918 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.377843    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82zsx\" (UniqueName: \"kubernetes.io/projected/50dad950-53be-49e3-883b-d803ac7858df-kube-api-access-82zsx\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:06.377918 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.377867    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50dad950-53be-49e3-883b-d803ac7858df-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:06.462774 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.462739    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"]
Oct 31 19:41:06.464801 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:06.464779    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3e4a2c-8849c7c77-7bnqz"]
Oct 31 19:41:07.201635 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:07.201559    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"
Oct 31 19:41:07.201635 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:07.201559    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws" event={"ID":"50dad950-53be-49e3-883b-d803ac7858df","Type":"ContainerDied","Data":"024ea2734113e4ab2dfca833724b9363dc7a88d52c1bae7c7419d60fe3247324"}
Oct 31 19:41:07.202170 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:07.201692    4484 scope.go:122] "RemoveContainer" containerID="bc2a0f564dec4353e855a72ef2421569b70401cda30ff554c8df100e5d991a64"
Oct 31 19:41:07.239375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:07.239342    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:07.372092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:07.372062    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"]
Oct 31 19:41:07.375770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:07.375747    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3e4a2c-754488f95f-smfws"]
Oct 31 19:41:07.993388 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:07.993353    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1142e654-6da9-46c3-a10e-c84bdd017f78" path="/var/lib/kubelet/pods/1142e654-6da9-46c3-a10e-c84bdd017f78/volumes"
Oct 31 19:41:07.993822 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:07.993806    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50dad950-53be-49e3-883b-d803ac7858df" path="/var/lib/kubelet/pods/50dad950-53be-49e3-883b-d803ac7858df/volumes"
Oct 31 19:41:08.122028 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:08.121990    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:10.597240 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:10.597207    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:10.993697 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:10.993579    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:11.392591 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:11.392495    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:11.863670 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:11.863642    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:12.155382 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:12.155294    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:15.634639 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:15.634601    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:16.230210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:16.230159    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:16.538509 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:16.538461    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:20.547908 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:20.547873    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:24.723281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:24.723235    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:25.467052 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:41:25.467014    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:41:26.423978 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:26.423943    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:28.237936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:28.237905    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:41:31.281733 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:31.281696    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:32.196401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:32.196370    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:34.095804 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.095774    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"]
Oct 31 19:41:34.096282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096268    4484 cpu_manager.go:414] "containerMap: removing container" podUID="020ad937-bef8-41ae-9394-eab2f30eefba" containerName="main"
Oct 31 19:41:34.096331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096284    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020ad937-bef8-41ae-9394-eab2f30eefba" containerName="main"
Oct 31 19:41:34.096331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096298    4484 cpu_manager.go:414] "containerMap: removing container" podUID="3b6b07cc-6154-4582-8e24-fc085949657a" containerName="main"
Oct 31 19:41:34.096331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096303    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6b07cc-6154-4582-8e24-fc085949657a" containerName="main"
Oct 31 19:41:34.096331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096313    4484 cpu_manager.go:414] "containerMap: removing container" podUID="24073104-4724-49d2-8056-64b9ffc3870b" containerName="maas-api"
Oct 31 19:41:34.096331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096317    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24073104-4724-49d2-8056-64b9ffc3870b" containerName="maas-api"
Oct 31 19:41:34.096331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096325    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0e5e6ede-697c-47c4-b53b-99a417db3f2d" containerName="payload-pre-processing"
Oct 31 19:41:34.096331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096329    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5e6ede-697c-47c4-b53b-99a417db3f2d" containerName="payload-pre-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096336    4484 cpu_manager.go:414] "containerMap: removing container" podUID="65891533-3ba8-44da-b413-d6936580f382" containerName="payload-pre-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096349    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65891533-3ba8-44da-b413-d6936580f382" containerName="payload-pre-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096362    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096366    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096372    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" containerName="cleanup"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096376    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" containerName="cleanup"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096383    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a91dc442-830a-4436-bab0-fedc94f75038" containerName="main"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096387    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91dc442-830a-4436-bab0-fedc94f75038" containerName="main"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096395    4484 cpu_manager.go:414] "containerMap: removing container" podUID="b3a0d3d9-238f-44e4-9894-11feed845de8" containerName="payload-pre-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096399    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a0d3d9-238f-44e4-9894-11feed845de8" containerName="payload-pre-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096405    4484 cpu_manager.go:414] "containerMap: removing container" podUID="7fdc08a1-c720-4038-acca-b931fbec7633" containerName="main"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096409    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fdc08a1-c720-4038-acca-b931fbec7633" containerName="main"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096418    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5410d4d5-bc30-456d-a17b-65e13b084937" containerName="payload-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096422    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5410d4d5-bc30-456d-a17b-65e13b084937" containerName="payload-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096428    4484 cpu_manager.go:414] "containerMap: removing container" podUID="1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a" containerName="payload-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096431    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a" containerName="payload-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096444    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096448    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096456    4484 cpu_manager.go:414] "containerMap: removing container" podUID="50dad950-53be-49e3-883b-d803ac7858df" containerName="payload-pre-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096463    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50dad950-53be-49e3-883b-d803ac7858df" containerName="payload-pre-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096482    4484 cpu_manager.go:414] "containerMap: removing container" podUID="307a9ca2-cb52-4131-b750-2fb5d89e8235" containerName="main"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096490    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307a9ca2-cb52-4131-b750-2fb5d89e8235" containerName="main"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096496    4484 cpu_manager.go:414] "containerMap: removing container" podUID="7d25c45a-e75d-4299-a206-c3875e851f43" containerName="maas-api"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096500    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d25c45a-e75d-4299-a206-c3875e851f43" containerName="maas-api"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096514    4484 cpu_manager.go:414] "containerMap: removing container" podUID="7f6a2a64-1451-4acc-8caf-17f9b16c30e8" containerName="payload-pre-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096519    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f6a2a64-1451-4acc-8caf-17f9b16c30e8" containerName="payload-pre-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096525    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c3ee3c68-0043-49ac-be93-fccb0cb0f932" containerName="istio-proxy"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096529    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ee3c68-0043-49ac-be93-fccb0cb0f932" containerName="istio-proxy"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096534    4484 cpu_manager.go:414] "containerMap: removing container" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerName="istio-proxy"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096538    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerName="istio-proxy"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096543    4484 cpu_manager.go:414] "containerMap: removing container" podUID="169ec18f-05f8-4fec-a540-9ade4b14863d" containerName="payload-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096548    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169ec18f-05f8-4fec-a540-9ade4b14863d" containerName="payload-processing"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096554    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a901ad86-48e1-422b-a342-64e3433fd101" containerName="cleanup"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096558    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a901ad86-48e1-422b-a342-64e3433fd101" containerName="cleanup"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096571    4484 cpu_manager.go:414] "containerMap: removing container" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" containerName="cleanup"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096575    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" containerName="cleanup"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096587    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb" containerName="cleanup"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096591    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb" containerName="cleanup"
Oct 31 19:41:34.096580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096598    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c010d33f-79f2-4396-83aa-ef85d00d4c81" containerName="payload-pre-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096602    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c010d33f-79f2-4396-83aa-ef85d00d4c81" containerName="payload-pre-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096607    4484 cpu_manager.go:414] "containerMap: removing container" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerName="istio-proxy"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096611    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerName="istio-proxy"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096621    4484 cpu_manager.go:414] "containerMap: removing container" podUID="73b43632-5dbb-4491-b1b8-48c1383f7266" containerName="payload-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096624    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b43632-5dbb-4491-b1b8-48c1383f7266" containerName="payload-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096630    4484 cpu_manager.go:414] "containerMap: removing container" podUID="1142e654-6da9-46c3-a10e-c84bdd017f78" containerName="payload-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096633    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1142e654-6da9-46c3-a10e-c84bdd017f78" containerName="payload-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096713    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c010d33f-79f2-4396-83aa-ef85d00d4c81" containerName="payload-pre-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096722    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerName="istio-proxy"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096728    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b43632-5dbb-4491-b1b8-48c1383f7266" containerName="payload-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096734    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1142e654-6da9-46c3-a10e-c84bdd017f78" containerName="payload-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096740    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73492d9-5852-498a-a6b7-10e2f446feeb" containerName="cleanup"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096745    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="020ad937-bef8-41ae-9394-eab2f30eefba" containerName="main"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096750    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6b07cc-6154-4582-8e24-fc085949657a" containerName="main"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096754    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="24073104-4724-49d2-8056-64b9ffc3870b" containerName="maas-api"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096758    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5e6ede-697c-47c4-b53b-99a417db3f2d" containerName="payload-pre-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096764    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="65891533-3ba8-44da-b413-d6936580f382" containerName="payload-pre-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096768    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbcf347-7edf-499d-87ea-4ea645f2631e" containerName="istio-proxy"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096773    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bddf03c-e580-4a67-b122-2fb0ebb7b7ac" containerName="cleanup"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096778    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91dc442-830a-4436-bab0-fedc94f75038" containerName="main"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096782    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0f8eb1-6091-43f3-8928-9b06bad2daae" containerName="cleanup"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096787    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a0d3d9-238f-44e4-9894-11feed845de8" containerName="payload-pre-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096791    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fdc08a1-c720-4038-acca-b931fbec7633" containerName="main"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096800    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5410d4d5-bc30-456d-a17b-65e13b084937" containerName="payload-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096804    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff4f7c3-15e2-4cf5-8c7b-d02ce9bbdf3a" containerName="payload-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096808    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d62a86e-a9fd-403b-92c8-b9617f803b85" containerName="istio-proxy"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096814    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="50dad950-53be-49e3-883b-d803ac7858df" containerName="payload-pre-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096818    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="307a9ca2-cb52-4131-b750-2fb5d89e8235" containerName="main"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096827    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d25c45a-e75d-4299-a206-c3875e851f43" containerName="maas-api"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096833    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f6a2a64-1451-4acc-8caf-17f9b16c30e8" containerName="payload-pre-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096838    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ee3c68-0043-49ac-be93-fccb0cb0f932" containerName="istio-proxy"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096843    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="169ec18f-05f8-4fec-a540-9ade4b14863d" containerName="payload-processing"
Oct 31 19:41:34.097779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.096848    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a901ad86-48e1-422b-a342-64e3433fd101" containerName="cleanup"
Oct 31 19:41:34.101795 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.101780    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.157840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.157811    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"]
Oct 31 19:41:34.178483 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.178437    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:34.206106 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.206051    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-0f794f\"/\"openshift-service-ca.crt\""
Oct 31 19:41:34.206106 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.206076    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-0f794f\"/\"default-dockercfg-t4r6f\""
Oct 31 19:41:34.206106 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.206064    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-0f794f\"/\"kube-root-ca.crt\""
Oct 31 19:41:34.206106 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.206055    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-0f794f\"/\"rate-test-model-0f794f-kserve-self-signed-certs\""
Oct 31 19:41:34.266962 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.266924    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d0ffbc6-082f-48d3-8d04-e055df568533-tls-certs\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.267148 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.267011    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-dshm\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.267148 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.267045    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-home\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.267148 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.267080    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgm84\" (UniqueName: \"kubernetes.io/projected/9d0ffbc6-082f-48d3-8d04-e055df568533-kube-api-access-cgm84\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.267325 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.267207    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-tmp-dir\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.267865 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.267834    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-model-cache\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.368880 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.368794    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-dshm\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.368880 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.368832    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-home\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.368880 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.368854    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgm84\" (UniqueName: \"kubernetes.io/projected/9d0ffbc6-082f-48d3-8d04-e055df568533-kube-api-access-cgm84\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.369173 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.368888    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-tmp-dir\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.369173 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.368919    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-model-cache\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.369173 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.369064    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d0ffbc6-082f-48d3-8d04-e055df568533-tls-certs\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.369375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.369259    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-home\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.369445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.369373    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-model-cache\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.369535 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.369447    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-tmp-dir\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.371277 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.371257    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-dshm\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.371603 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.371584    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d0ffbc6-082f-48d3-8d04-e055df568533-tls-certs\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.429053 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.429023    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgm84\" (UniqueName: \"kubernetes.io/projected/9d0ffbc6-082f-48d3-8d04-e055df568533-kube-api-access-cgm84\") pod \"rate-test-model-0f794f-kserve-65dc658d69-zgz54\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") " pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:34.714359 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:34.714275    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:35.002421 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.002371    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:35.088625 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.088594    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"]
Oct 31 19:41:35.090293 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:41:35.090266    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d0ffbc6_082f_48d3_8d04_e055df568533.slice/crio-12f56ca12c0206361d8fda6c34ac1fe0e1c9dfc6c167e88c21cce546ac7bb1c7 WatchSource:0}: Error finding container 12f56ca12c0206361d8fda6c34ac1fe0e1c9dfc6c167e88c21cce546ac7bb1c7: Status 404 returned error can't find the container with id 12f56ca12c0206361d8fda6c34ac1fe0e1c9dfc6c167e88c21cce546ac7bb1c7
Oct 31 19:41:35.272727 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.272635    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:35.349703 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.349665    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54" event={"ID":"9d0ffbc6-082f-48d3-8d04-e055df568533","Type":"ContainerStarted","Data":"44ebdaae49f6bc9796154303ec6fbf573833ac37684099b92e0942ad3e9f3f11"}
Oct 31 19:41:35.349870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.349711    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54" event={"ID":"9d0ffbc6-082f-48d3-8d04-e055df568533","Type":"ContainerStarted","Data":"12f56ca12c0206361d8fda6c34ac1fe0e1c9dfc6c167e88c21cce546ac7bb1c7"}
Oct 31 19:41:35.468711 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.468656    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54" podStartSLOduration=1.468642449 podStartE2EDuration="1.468642449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:35.466738407 +0000 UTC m=+4841.613188271" watchObservedRunningTime="2026-10-31 19:41:35.468642449 +0000 UTC m=+4841.615092326"
Oct 31 19:41:35.640387 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.640294    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7560-76b6ef"]
Oct 31 19:41:35.640982 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.640960    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b435e8f9-377a-4d33-bee0-a5e03e4eb4d7" containerName="istio-proxy"
Oct 31 19:41:35.644166 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.644146    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-76b6ef"
Oct 31 19:41:35.700857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.700831    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-76b6ef"]
Oct 31 19:41:35.714768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.714729    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:35.783635 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.783606    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v4sv\" (UniqueName: \"kubernetes.io/projected/3ab0658e-c7ff-48fc-aa43-fd8c53569a12-kube-api-access-4v4sv\") pod \"test-curl-7560-76b6ef\" (UID: \"3ab0658e-c7ff-48fc-aa43-fd8c53569a12\") " pod="opendatahub/test-curl-7560-76b6ef"
Oct 31 19:41:35.787370 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.787348    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t9bg2\""
Oct 31 19:41:35.884934 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.884899    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v4sv\" (UniqueName: \"kubernetes.io/projected/3ab0658e-c7ff-48fc-aa43-fd8c53569a12-kube-api-access-4v4sv\") pod \"test-curl-7560-76b6ef\" (UID: \"3ab0658e-c7ff-48fc-aa43-fd8c53569a12\") " pod="opendatahub/test-curl-7560-76b6ef"
Oct 31 19:41:35.943688 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.943619    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v4sv\" (UniqueName: \"kubernetes.io/projected/3ab0658e-c7ff-48fc-aa43-fd8c53569a12-kube-api-access-4v4sv\") pod \"test-curl-7560-76b6ef\" (UID: \"3ab0658e-c7ff-48fc-aa43-fd8c53569a12\") " pod="opendatahub/test-curl-7560-76b6ef"
Oct 31 19:41:35.956561 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:35.956533    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-76b6ef"
Oct 31 19:41:36.081019 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:36.080979    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:36.328678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:36.328654    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-76b6ef"]
Oct 31 19:41:36.330623 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:41:36.330595    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ab0658e_c7ff_48fc_aa43_fd8c53569a12.slice/crio-9d71028e4c730622a2da0f1439e574c0d9e6ff20a35eeab2eeae03d025e8c4b8 WatchSource:0}: Error finding container 9d71028e4c730622a2da0f1439e574c0d9e6ff20a35eeab2eeae03d025e8c4b8: Status 404 returned error can't find the container with id 9d71028e4c730622a2da0f1439e574c0d9e6ff20a35eeab2eeae03d025e8c4b8
Oct 31 19:41:36.355863 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:36.355836    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-76b6ef" event={"ID":"3ab0658e-c7ff-48fc-aa43-fd8c53569a12","Type":"ContainerStarted","Data":"9d71028e4c730622a2da0f1439e574c0d9e6ff20a35eeab2eeae03d025e8c4b8"}
Oct 31 19:41:36.649263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:36.649173    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:37.175563 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:37.175410    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:37.187981 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:37.187956    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"]
Oct 31 19:41:37.188157 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:37.188134    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6" podUID="d97d19fe-d536-42cd-8529-39d26421d668" containerName="istio-proxy" containerID="cri-o://c54bc2f2469f85bcacd086d689e922083d3b14f4082faa6a9fb19e862d09e53c" gracePeriod=30
Oct 31 19:41:37.362928 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:37.362891    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-76b6ef" event={"ID":"3ab0658e-c7ff-48fc-aa43-fd8c53569a12","Type":"ContainerStarted","Data":"fef663dc4a708fedd1787fd3db4012a2063ea79fcff30c869d3c9b8e5e2d320a"}
Oct 31 19:41:37.484926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:37.484832    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7560-76b6ef" podStartSLOduration=2.484818214 podStartE2EDuration="2.484818214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:37.484448249 +0000 UTC m=+4843.630898120" watchObservedRunningTime="2026-10-31 19:41:37.484818214 +0000 UTC m=+4843.631268129"
Oct 31 19:41:38.397434 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:38.397394    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:38.686966 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:38.686890    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:38.989536 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:38.989508    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7560-76b6ef"]
Oct 31 19:41:38.992709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:38.992685    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7560-76b6ef"]
Oct 31 19:41:39.020950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.020922    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:39.131550 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.131521    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7560-37ebcf"]
Oct 31 19:41:39.131986 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.131973    4484 cpu_manager.go:414] "containerMap: removing container" podUID="3ab0658e-c7ff-48fc-aa43-fd8c53569a12" containerName="test-curl-7560-76b6ef"
Oct 31 19:41:39.132033 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.131987    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab0658e-c7ff-48fc-aa43-fd8c53569a12" containerName="test-curl-7560-76b6ef"
Oct 31 19:41:39.132095 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.132087    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab0658e-c7ff-48fc-aa43-fd8c53569a12" containerName="test-curl-7560-76b6ef"
Oct 31 19:41:39.135266 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.135249    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-37ebcf"
Oct 31 19:41:39.191366 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.191342    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-37ebcf"]
Oct 31 19:41:39.215864 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.215831    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6tcd\" (UniqueName: \"kubernetes.io/projected/4554da1d-ba14-44f4-9869-7e6e2f9113a9-kube-api-access-t6tcd\") pod \"test-curl-7560-37ebcf\" (UID: \"4554da1d-ba14-44f4-9869-7e6e2f9113a9\") " pod="opendatahub/test-curl-7560-37ebcf"
Oct 31 19:41:39.316696 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.316624    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tcd\" (UniqueName: \"kubernetes.io/projected/4554da1d-ba14-44f4-9869-7e6e2f9113a9-kube-api-access-t6tcd\") pod \"test-curl-7560-37ebcf\" (UID: \"4554da1d-ba14-44f4-9869-7e6e2f9113a9\") " pod="opendatahub/test-curl-7560-37ebcf"
Oct 31 19:41:39.374420 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.374367    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7560-76b6ef" podUID="3ab0658e-c7ff-48fc-aa43-fd8c53569a12" containerName="test-curl-7560-76b6ef" containerID="cri-o://fef663dc4a708fedd1787fd3db4012a2063ea79fcff30c869d3c9b8e5e2d320a" gracePeriod=2
Oct 31 19:41:39.375210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.375190    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6tcd\" (UniqueName: \"kubernetes.io/projected/4554da1d-ba14-44f4-9869-7e6e2f9113a9-kube-api-access-t6tcd\") pod \"test-curl-7560-37ebcf\" (UID: \"4554da1d-ba14-44f4-9869-7e6e2f9113a9\") " pod="opendatahub/test-curl-7560-37ebcf"
Oct 31 19:41:39.426188 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:41:39.426152    4484 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7560-76b6ef\" is forbidden: User \"system:node:ip-10-0-142-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-120.us-west-1.compute.internal' and this object" podUID="3ab0658e-c7ff-48fc-aa43-fd8c53569a12" pod="opendatahub/test-curl-7560-76b6ef"
Oct 31 19:41:39.447767 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.447746    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-37ebcf"
Oct 31 19:41:39.819739 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:39.819714    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7560-37ebcf"]
Oct 31 19:41:39.820628 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:41:39.820605    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4554da1d_ba14_44f4_9869_7e6e2f9113a9.slice/crio-c750fcd146866d65bd615ab900d29cdbd8762bfee5f1207c7abc73d6bf5eeeb4 WatchSource:0}: Error finding container c750fcd146866d65bd615ab900d29cdbd8762bfee5f1207c7abc73d6bf5eeeb4: Status 404 returned error can't find the container with id c750fcd146866d65bd615ab900d29cdbd8762bfee5f1207c7abc73d6bf5eeeb4
Oct 31 19:41:40.383189 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:40.383154    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-37ebcf" event={"ID":"4554da1d-ba14-44f4-9869-7e6e2f9113a9","Type":"ContainerStarted","Data":"f668ebec4a896a2058ad3a99dbcdc2f38cbf339fbd996fb32619ef505c3b7488"}
Oct 31 19:41:40.383189 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:40.383198    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7560-37ebcf" event={"ID":"4554da1d-ba14-44f4-9869-7e6e2f9113a9","Type":"ContainerStarted","Data":"c750fcd146866d65bd615ab900d29cdbd8762bfee5f1207c7abc73d6bf5eeeb4"}
Oct 31 19:41:40.501051 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:40.501000    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7560-37ebcf" podStartSLOduration=1.500985876 podStartE2EDuration="1.500985876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:40.50004362 +0000 UTC m=+4846.646493484" watchObservedRunningTime="2026-10-31 19:41:40.500985876 +0000 UTC m=+4846.647435740"
Oct 31 19:41:41.422572 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:41.422545    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:41.606498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:41.606448    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-76b6ef"
Oct 31 19:41:41.640123 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:41.640049    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab0658e-c7ff-48fc-aa43-fd8c53569a12-kube-api-access-4v4sv\" (UniqueName: \"kubernetes.io/projected/3ab0658e-c7ff-48fc-aa43-fd8c53569a12-kube-api-access-4v4sv\") pod \"3ab0658e-c7ff-48fc-aa43-fd8c53569a12\" (UID: \"3ab0658e-c7ff-48fc-aa43-fd8c53569a12\") "
Oct 31 19:41:41.642232 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:41.642206    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab0658e-c7ff-48fc-aa43-fd8c53569a12-kube-api-access-4v4sv" pod "3ab0658e-c7ff-48fc-aa43-fd8c53569a12" (UID: "3ab0658e-c7ff-48fc-aa43-fd8c53569a12"). InnerVolumeSpecName "kube-api-access-4v4sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:41.741283 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:41.741245    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v4sv\" (UniqueName: \"kubernetes.io/projected/3ab0658e-c7ff-48fc-aa43-fd8c53569a12-kube-api-access-4v4sv\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:41.914932 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:41.914846    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7560-37ebcf"]
Oct 31 19:41:41.919916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:41.919886    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7560-37ebcf"]
Oct 31 19:41:41.993523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:41.993495    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ab0658e-c7ff-48fc-aa43-fd8c53569a12" path="/var/lib/kubelet/pods/3ab0658e-c7ff-48fc-aa43-fd8c53569a12/volumes"
Oct 31 19:41:42.396414 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.396384    4484 generic.go:356] "Generic (PLEG): container finished" podID="d97d19fe-d536-42cd-8529-39d26421d668" containerID="c54bc2f2469f85bcacd086d689e922083d3b14f4082faa6a9fb19e862d09e53c" exitCode=0
Oct 31 19:41:42.396635 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.396455    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6" event={"ID":"d97d19fe-d536-42cd-8529-39d26421d668","Type":"ContainerDied","Data":"c54bc2f2469f85bcacd086d689e922083d3b14f4082faa6a9fb19e862d09e53c"}
Oct 31 19:41:42.397850 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.397823    4484 generic.go:356] "Generic (PLEG): container finished" podID="3ab0658e-c7ff-48fc-aa43-fd8c53569a12" containerID="fef663dc4a708fedd1787fd3db4012a2063ea79fcff30c869d3c9b8e5e2d320a" exitCode=137
Oct 31 19:41:42.397976 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.397871    4484 scope.go:122] "RemoveContainer" containerID="fef663dc4a708fedd1787fd3db4012a2063ea79fcff30c869d3c9b8e5e2d320a"
Oct 31 19:41:42.397976 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.397889    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-76b6ef"
Oct 31 19:41:42.398118 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.398096    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7560-37ebcf" podUID="4554da1d-ba14-44f4-9869-7e6e2f9113a9" containerName="test-curl-7560-37ebcf" containerID="cri-o://f668ebec4a896a2058ad3a99dbcdc2f38cbf339fbd996fb32619ef505c3b7488" gracePeriod=2
Oct 31 19:41:42.407899 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.407879    4484 scope.go:122] "RemoveContainer" containerID="fef663dc4a708fedd1787fd3db4012a2063ea79fcff30c869d3c9b8e5e2d320a"
Oct 31 19:41:42.408206 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:41:42.408185    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef663dc4a708fedd1787fd3db4012a2063ea79fcff30c869d3c9b8e5e2d320a\": container with ID starting with fef663dc4a708fedd1787fd3db4012a2063ea79fcff30c869d3c9b8e5e2d320a not found: ID does not exist" containerID="fef663dc4a708fedd1787fd3db4012a2063ea79fcff30c869d3c9b8e5e2d320a"
Oct 31 19:41:42.408286 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.408227    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef663dc4a708fedd1787fd3db4012a2063ea79fcff30c869d3c9b8e5e2d320a"} err="failed to get container status \"fef663dc4a708fedd1787fd3db4012a2063ea79fcff30c869d3c9b8e5e2d320a\": rpc error: code = NotFound desc = could not find container \"fef663dc4a708fedd1787fd3db4012a2063ea79fcff30c869d3c9b8e5e2d320a\": container with ID starting with fef663dc4a708fedd1787fd3db4012a2063ea79fcff30c869d3c9b8e5e2d320a not found: ID does not exist"
Oct 31 19:41:42.446889 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.446869    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 19:41:42.549277 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549250    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-envoy\") pod \"d97d19fe-d536-42cd-8529-39d26421d668\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") "
Oct 31 19:41:42.549459 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549289    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-kube-api-access-vz4cm\" (UniqueName: \"kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-kube-api-access-vz4cm\") pod \"d97d19fe-d536-42cd-8529-39d26421d668\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") "
Oct 31 19:41:42.549459 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549320    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-credential-socket\") pod \"d97d19fe-d536-42cd-8529-39d26421d668\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") "
Oct 31 19:41:42.549459 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549398    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-certs\") pod \"d97d19fe-d536-42cd-8529-39d26421d668\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") "
Oct 31 19:41:42.549459 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549442    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-data\") pod \"d97d19fe-d536-42cd-8529-39d26421d668\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") "
Oct 31 19:41:42.549741 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549510    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d97d19fe-d536-42cd-8529-39d26421d668-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97d19fe-d536-42cd-8529-39d26421d668-istio-podinfo\") pod \"d97d19fe-d536-42cd-8529-39d26421d668\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") "
Oct 31 19:41:42.549741 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549536    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-istio-token\" (UniqueName: \"kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-istio-token\") pod \"d97d19fe-d536-42cd-8529-39d26421d668\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") "
Oct 31 19:41:42.549741 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549572    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d97d19fe-d536-42cd-8529-39d26421d668-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97d19fe-d536-42cd-8529-39d26421d668-istiod-ca-cert\") pod \"d97d19fe-d536-42cd-8529-39d26421d668\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") "
Oct 31 19:41:42.549741 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549617    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-socket\") pod \"d97d19fe-d536-42cd-8529-39d26421d668\" (UID: \"d97d19fe-d536-42cd-8529-39d26421d668\") "
Oct 31 19:41:42.549741 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549718    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-certs" pod "d97d19fe-d536-42cd-8529-39d26421d668" (UID: "d97d19fe-d536-42cd-8529-39d26421d668"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:42.549741 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549734    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-credential-socket" pod "d97d19fe-d536-42cd-8529-39d26421d668" (UID: "d97d19fe-d536-42cd-8529-39d26421d668"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:42.550220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549929    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-socket" pod "d97d19fe-d536-42cd-8529-39d26421d668" (UID: "d97d19fe-d536-42cd-8529-39d26421d668"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:42.550220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.549947    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-data" pod "d97d19fe-d536-42cd-8529-39d26421d668" (UID: "d97d19fe-d536-42cd-8529-39d26421d668"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:42.550220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.550015    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97d19fe-d536-42cd-8529-39d26421d668-istiod-ca-cert" pod "d97d19fe-d536-42cd-8529-39d26421d668" (UID: "d97d19fe-d536-42cd-8529-39d26421d668"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:42.550394 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.550217    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:42.550394 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.550255    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:42.550394 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.550270    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:42.550394 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.550283    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:42.551715 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.551693    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-envoy" pod "d97d19fe-d536-42cd-8529-39d26421d668" (UID: "d97d19fe-d536-42cd-8529-39d26421d668"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:42.551797 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.551719    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-kube-api-access-vz4cm" pod "d97d19fe-d536-42cd-8529-39d26421d668" (UID: "d97d19fe-d536-42cd-8529-39d26421d668"). InnerVolumeSpecName "kube-api-access-vz4cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:42.551866 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.551812    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-istio-token" pod "d97d19fe-d536-42cd-8529-39d26421d668" (UID: "d97d19fe-d536-42cd-8529-39d26421d668"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:42.552163 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.552143    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d97d19fe-d536-42cd-8529-39d26421d668-istio-podinfo" pod "d97d19fe-d536-42cd-8529-39d26421d668" (UID: "d97d19fe-d536-42cd-8529-39d26421d668"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:42.651832 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.651799    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97d19fe-d536-42cd-8529-39d26421d668-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:42.651832 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.651827    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:42.651832 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.651840    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97d19fe-d536-42cd-8529-39d26421d668-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:42.652257 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.651848    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97d19fe-d536-42cd-8529-39d26421d668-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:42.652257 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:42.651858    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz4cm\" (UniqueName: \"kubernetes.io/projected/d97d19fe-d536-42cd-8529-39d26421d668-kube-api-access-vz4cm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:43.405602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:43.405566    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6" event={"ID":"d97d19fe-d536-42cd-8529-39d26421d668","Type":"ContainerDied","Data":"cedc8cb6d313fdfe269c0ec2658d80b71b3746b3516d10aeb7de54ea9ca42c8d"}
Oct 31 19:41:43.405602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:43.405589    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"
Oct 31 19:41:43.405818 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:43.405634    4484 scope.go:122] "RemoveContainer" containerID="c54bc2f2469f85bcacd086d689e922083d3b14f4082faa6a9fb19e862d09e53c"
Oct 31 19:41:43.572334 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:43.572302    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"]
Oct 31 19:41:43.623396 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:43.623363    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e4a2c-openshift-default-7f8d64b6cf-qhmx6"]
Oct 31 19:41:44.000722 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.000688    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d97d19fe-d536-42cd-8529-39d26421d668" path="/var/lib/kubelet/pods/d97d19fe-d536-42cd-8529-39d26421d668/volumes"
Oct 31 19:41:44.642940 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.642917    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-37ebcf"
Oct 31 19:41:44.667276 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.667216    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"]
Oct 31 19:41:44.667706 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.667693    4484 cpu_manager.go:414] "containerMap: removing container" podUID="4554da1d-ba14-44f4-9869-7e6e2f9113a9" containerName="test-curl-7560-37ebcf"
Oct 31 19:41:44.667761 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.667707    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4554da1d-ba14-44f4-9869-7e6e2f9113a9" containerName="test-curl-7560-37ebcf"
Oct 31 19:41:44.667761 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.667733    4484 cpu_manager.go:414] "containerMap: removing container" podUID="d97d19fe-d536-42cd-8529-39d26421d668" containerName="istio-proxy"
Oct 31 19:41:44.667761 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.667738    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d97d19fe-d536-42cd-8529-39d26421d668" containerName="istio-proxy"
Oct 31 19:41:44.667878 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.667820    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d97d19fe-d536-42cd-8529-39d26421d668" containerName="istio-proxy"
Oct 31 19:41:44.667878 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.667832    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="4554da1d-ba14-44f4-9869-7e6e2f9113a9" containerName="test-curl-7560-37ebcf"
Oct 31 19:41:44.670932 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.670915    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.673138 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.673117    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4554da1d-ba14-44f4-9869-7e6e2f9113a9-kube-api-access-t6tcd\" (UniqueName: \"kubernetes.io/projected/4554da1d-ba14-44f4-9869-7e6e2f9113a9-kube-api-access-t6tcd\") pod \"4554da1d-ba14-44f4-9869-7e6e2f9113a9\" (UID: \"4554da1d-ba14-44f4-9869-7e6e2f9113a9\") "
Oct 31 19:41:44.675165 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.675146    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4554da1d-ba14-44f4-9869-7e6e2f9113a9-kube-api-access-t6tcd" pod "4554da1d-ba14-44f4-9869-7e6e2f9113a9" (UID: "4554da1d-ba14-44f4-9869-7e6e2f9113a9"). InnerVolumeSpecName "kube-api-access-t6tcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:44.715017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.714996    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:44.724180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.724156    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6cfd68-openshift-default-dockercfg-bfq85\""
Oct 31 19:41:44.727621 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.727598    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:44.774367 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.774336    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-socket\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.774537 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.774379    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-envoy\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.774612 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.774540    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-podinfo\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.774731 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.774714    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9z5z\" (UniqueName: \"kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-kube-api-access-h9z5z\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.774788 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.774751    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-token\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.774856 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.774844    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-certs\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.774889 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.774875    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-data\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.774995 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.774982    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-credential-socket\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.775059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.775005    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istiod-ca-cert\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.775059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.775042    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6tcd\" (UniqueName: \"kubernetes.io/projected/4554da1d-ba14-44f4-9869-7e6e2f9113a9-kube-api-access-t6tcd\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:41:44.777824 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.777807    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"]
Oct 31 19:41:44.875647 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.875618    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-socket\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.875830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.875659    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-envoy\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.875830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.875707    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-podinfo\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.875830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.875726    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9z5z\" (UniqueName: \"kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-kube-api-access-h9z5z\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.875830 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.875747    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-token\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.876059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.875888    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-certs\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.876059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.875951    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-data\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.876059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.875996    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-credential-socket\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.876059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.876027    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istiod-ca-cert\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.876239 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.876053    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-socket\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.876288 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.876271    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-certs\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.876533 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.876430    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-credential-socket\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.876652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.876629    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-data\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.876700 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.876647    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istiod-ca-cert\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.878173 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.878152    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-envoy\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.878341 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.878323    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-podinfo\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.933907 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.933841    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-token\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.934015 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.933915    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9z5z\" (UniqueName: \"kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-kube-api-access-h9z5z\") pod \"e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") " pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:44.991028 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:44.991001    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:45.413559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:45.413531    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"]
Oct 31 19:41:45.414680 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:41:45.414646    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fc3a9e_624f_4971_97e8_4a4e3ff08181.slice/crio-e6cd72b81dfe746a99b744a9e9895312bd132189ab6716249114bbff8467af7c WatchSource:0}: Error finding container e6cd72b81dfe746a99b744a9e9895312bd132189ab6716249114bbff8467af7c: Status 404 returned error can't find the container with id e6cd72b81dfe746a99b744a9e9895312bd132189ab6716249114bbff8467af7c
Oct 31 19:41:45.417092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:45.417062    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:41:45.417168 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:45.417125    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:41:45.417168 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:45.417156    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:41:45.419159 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:45.419093    4484 generic.go:356] "Generic (PLEG): container finished" podID="4554da1d-ba14-44f4-9869-7e6e2f9113a9" containerID="f668ebec4a896a2058ad3a99dbcdc2f38cbf339fbd996fb32619ef505c3b7488" exitCode=137
Oct 31 19:41:45.419159 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:45.419142    4484 scope.go:122] "RemoveContainer" containerID="f668ebec4a896a2058ad3a99dbcdc2f38cbf339fbd996fb32619ef505c3b7488"
Oct 31 19:41:45.419327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:45.419180    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7560-37ebcf"
Oct 31 19:41:45.436258 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:45.436237    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:41:45.445933 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:45.445795    4484 scope.go:122] "RemoveContainer" containerID="f668ebec4a896a2058ad3a99dbcdc2f38cbf339fbd996fb32619ef505c3b7488"
Oct 31 19:41:45.446121 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:41:45.446101    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f668ebec4a896a2058ad3a99dbcdc2f38cbf339fbd996fb32619ef505c3b7488\": container with ID starting with f668ebec4a896a2058ad3a99dbcdc2f38cbf339fbd996fb32619ef505c3b7488 not found: ID does not exist" containerID="f668ebec4a896a2058ad3a99dbcdc2f38cbf339fbd996fb32619ef505c3b7488"
Oct 31 19:41:45.446172 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:45.446131    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f668ebec4a896a2058ad3a99dbcdc2f38cbf339fbd996fb32619ef505c3b7488"} err="failed to get container status \"f668ebec4a896a2058ad3a99dbcdc2f38cbf339fbd996fb32619ef505c3b7488\": rpc error: code = NotFound desc = could not find container \"f668ebec4a896a2058ad3a99dbcdc2f38cbf339fbd996fb32619ef505c3b7488\": container with ID starting with f668ebec4a896a2058ad3a99dbcdc2f38cbf339fbd996fb32619ef505c3b7488 not found: ID does not exist"
Oct 31 19:41:45.622167 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:41:45.622122    4484 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7560-37ebcf\" is forbidden: User \"system:node:ip-10-0-142-120.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-120.us-west-1.compute.internal' and this object" podUID="4554da1d-ba14-44f4-9869-7e6e2f9113a9" pod="opendatahub/test-curl-7560-37ebcf"
Oct 31 19:41:45.993930 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:45.993895    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4554da1d-ba14-44f4-9869-7e6e2f9113a9" path="/var/lib/kubelet/pods/4554da1d-ba14-44f4-9869-7e6e2f9113a9/volumes"
Oct 31 19:41:46.136346 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:46.136313    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:46.426818 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:46.426723    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh" event={"ID":"d0fc3a9e-624f-4971-97e8-4a4e3ff08181","Type":"ContainerStarted","Data":"87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa"}
Oct 31 19:41:46.426818 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:46.426783    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh" event={"ID":"d0fc3a9e-624f-4971-97e8-4a4e3ff08181","Type":"ContainerStarted","Data":"e6cd72b81dfe746a99b744a9e9895312bd132189ab6716249114bbff8467af7c"}
Oct 31 19:41:46.600982 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:46.600926    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh" podStartSLOduration=2.60091 podStartE2EDuration="2.60091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:46.598679066 +0000 UTC m=+4852.745128929" watchObservedRunningTime="2026-10-31 19:41:46.60091 +0000 UTC m=+4852.747359864"
Oct 31 19:41:46.991186 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:46.991151    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:46.993020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:46.992993    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 19:41:46.993160 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:46.993037    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh" podUID="d0fc3a9e-624f-4971-97e8-4a4e3ff08181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 19:41:47.991708 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:47.991678    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 19:41:47.992132 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:47.991737    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh" podUID="d0fc3a9e-624f-4971-97e8-4a4e3ff08181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 19:41:48.992209 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:48.992175    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 19:41:48.992699 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:48.992234    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh" podUID="d0fc3a9e-624f-4971-97e8-4a4e3ff08181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 19:41:49.991754 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:49.991718    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 19:41:49.991946 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:49.991770    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh" podUID="d0fc3a9e-624f-4971-97e8-4a4e3ff08181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 19:41:50.403816 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:50.403729    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:50.992140 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:50.992106    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 31 19:41:50.992327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:50.992174    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh" podUID="d0fc3a9e-624f-4971-97e8-4a4e3ff08181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 31 19:41:51.996374 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:51.996345    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:51.996896 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:51.996686    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:51.997456 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:51.997436    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:41:52.121046 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:52.121009    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:55.468239 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:41:55.468201    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:41:56.843621 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:56.843586    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:58.969097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:58.969066    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:59.292839 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:41:59.292796    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:00.575423 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:00.575389    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:01.115327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:01.115291    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:01.595845 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:01.595813    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:01.867161 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:01.867080    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:04.683428 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:04.683388    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:04.938665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:04.938585    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:05.023870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:05.023838    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:42:06.571207 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:06.571174    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"]
Oct 31 19:42:06.642100 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:06.642061    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:06.862321 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:06.862233    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:07.574935 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:07.574898    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:11.328920 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:11.328884    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:12.110298 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:12.110252    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:12.786054 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:12.786015    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:16.069884 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:16.069848    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:19.248115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:19.248074    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"]
Oct 31 19:42:19.278786 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:19.278749    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:19.595503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:19.595391    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:19.861410 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:19.861320    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:21.040379 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.040333    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:21.591600 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.591558    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k" podUID="0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab" containerName="main" containerID="cri-o://b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d" gracePeriod=45
Oct 31 19:42:21.848923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.848869    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 19:42:21.946432 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.946404    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-kube-api-access-8fmqv\" (UniqueName: \"kubernetes.io/projected/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-kube-api-access-8fmqv\") pod \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") "
Oct 31 19:42:21.946599 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.946559    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tmp-dir\") pod \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") "
Oct 31 19:42:21.946655 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.946631    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-dshm\") pod \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") "
Oct 31 19:42:21.946702 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.946675    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-model-cache\") pod \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") "
Oct 31 19:42:21.946803 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.946774    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tmp-dir" pod "0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab" (UID: "0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:21.946803 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.946792    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tls-certs\") pod \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") "
Oct 31 19:42:21.947016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.946822    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-home\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-home\") pod \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\" (UID: \"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab\") "
Oct 31 19:42:21.947016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.946915    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-model-cache" pod "0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab" (UID: "0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:21.947146 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.947075    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-home" pod "0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab" (UID: "0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:21.947146 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.947125    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:21.947146 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.947138    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:21.947262 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.947147    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:21.949090 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.949059    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tls-certs" pod "0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab" (UID: "0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:21.949259 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.949238    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-dshm" pod "0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab" (UID: "0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:21.949362 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:21.949345    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-kube-api-access-8fmqv" pod "0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab" (UID: "0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab"). InnerVolumeSpecName "kube-api-access-8fmqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:22.048236 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:22.048205    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:22.048236 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:22.048233    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:22.048236 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:22.048243    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fmqv\" (UniqueName: \"kubernetes.io/projected/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab-kube-api-access-8fmqv\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:22.626282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:22.626249    4484 generic.go:356] "Generic (PLEG): container finished" podID="0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab" containerID="b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d" exitCode=0
Oct 31 19:42:22.626453 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:22.626287    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"
Oct 31 19:42:22.626453 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:22.626324    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k" event={"ID":"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab","Type":"ContainerDied","Data":"b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d"}
Oct 31 19:42:22.626453 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:22.626367    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k" event={"ID":"0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab","Type":"ContainerDied","Data":"08d87a495bbe36aeca6a4ccd252cb652edf86ca0e983152390e53582f990244f"}
Oct 31 19:42:22.626453 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:22.626384    4484 scope.go:122] "RemoveContainer" containerID="b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d"
Oct 31 19:42:22.635436 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:22.635420    4484 scope.go:122] "RemoveContainer" containerID="b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d"
Oct 31 19:42:22.635686 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:42:22.635669    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d\": container with ID starting with b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d not found: ID does not exist" containerID="b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d"
Oct 31 19:42:22.635744 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:22.635695    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d"} err="failed to get container status \"b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d\": rpc error: code = NotFound desc = could not find container \"b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d\": container with ID starting with b0020c899ee44cc07d25ba388bbd2671e5da16822ec03807225c20e577a8b19d not found: ID does not exist"
Oct 31 19:42:22.795441 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:22.795402    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"]
Oct 31 19:42:22.846456 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:22.846428    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-6279f5/distinct-2-w4-6279f5-kserve-b7b5d7f7f-2zm5k"]
Oct 31 19:42:23.993097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:23.993064    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab" path="/var/lib/kubelet/pods/0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab/volumes"
Oct 31 19:42:24.258541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:24.258437    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk" podUID="461e65fc-44a3-437f-93a9-bccc4a4d45b4" containerName="payload-processing" containerID="cri-o://ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc" gracePeriod=25
Oct 31 19:42:25.086596 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:25.086553    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:25.467287 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:42:25.467212    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:42:25.933665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:25.933621    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:26.395593 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:26.395559    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:42:28.084922 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:28.084886    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk" podUID="461e65fc-44a3-437f-93a9-bccc4a4d45b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:42:30.556211 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.556159    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:30.707458 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.707422    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"]
Oct 31 19:42:30.707919 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.707906    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab" containerName="main"
Oct 31 19:42:30.707979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.707922    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab" containerName="main"
Oct 31 19:42:30.708036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.708027    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae7a3ad-06bd-40ff-a5c0-9aa7410a57ab" containerName="main"
Oct 31 19:42:30.712617 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.712595    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.737923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.737893    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-dshm\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.738075 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.737971    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-model-cache\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.738122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.738067    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-home\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.738122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.738111    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tls-certs\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.738254 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.738230    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9777d\" (UniqueName: \"kubernetes.io/projected/f9db77c4-3afe-4b27-b7f2-8268383dfadb-kube-api-access-9777d\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.738291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.738270    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tmp-dir\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.768727 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.768706    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"]
Oct 31 19:42:30.817534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.817444    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-0f0e4e\"/\"openshift-service-ca.crt\""
Oct 31 19:42:30.817534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.817446    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-0f0e4e\"/\"rate-test-model-0f0e4e-kserve-self-signed-certs\""
Oct 31 19:42:30.817534 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.817446    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-0f0e4e\"/\"kube-root-ca.crt\""
Oct 31 19:42:30.824738 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.824708    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-0f0e4e\"/\"default-dockercfg-qcjdp\""
Oct 31 19:42:30.839602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.839568    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-dshm\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.839900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.839697    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-model-cache\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.839900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.839737    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-home\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.839900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.839767    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tls-certs\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.839900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.839830    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9777d\" (UniqueName: \"kubernetes.io/projected/f9db77c4-3afe-4b27-b7f2-8268383dfadb-kube-api-access-9777d\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.839900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.839852    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tmp-dir\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.840170 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.840151    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-model-cache\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.840256 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.840235    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tmp-dir\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.840738 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.840707    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-home\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.843029 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.843011    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tls-certs\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.843155 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.843074    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-dshm\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.899885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.899851    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9777d\" (UniqueName: \"kubernetes.io/projected/f9db77c4-3afe-4b27-b7f2-8268383dfadb-kube-api-access-9777d\") pod \"rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") " pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:30.963279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:30.963243    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:31.025551 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:31.025520    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:31.406319 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:31.406288    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"]
Oct 31 19:42:31.407909 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:42:31.407883    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9db77c4_3afe_4b27_b7f2_8268383dfadb.slice/crio-c01d0478fd7ff0115776b0a4320a55594440208605f4761aed6b0c37825c1a71 WatchSource:0}: Error finding container c01d0478fd7ff0115776b0a4320a55594440208605f4761aed6b0c37825c1a71: Status 404 returned error can't find the container with id c01d0478fd7ff0115776b0a4320a55594440208605f4761aed6b0c37825c1a71
Oct 31 19:42:31.678607 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:31.678514    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n" event={"ID":"f9db77c4-3afe-4b27-b7f2-8268383dfadb","Type":"ContainerStarted","Data":"000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf"}
Oct 31 19:42:31.678607 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:31.678562    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n" event={"ID":"f9db77c4-3afe-4b27-b7f2-8268383dfadb","Type":"ContainerStarted","Data":"c01d0478fd7ff0115776b0a4320a55594440208605f4761aed6b0c37825c1a71"}
Oct 31 19:42:31.797302 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:31.797254    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n" podStartSLOduration=1.797238399 podStartE2EDuration="1.797238399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:31.79664675 +0000 UTC m=+4897.943096614" watchObservedRunningTime="2026-10-31 19:42:31.797238399 +0000 UTC m=+4897.943688263"
Oct 31 19:42:32.026589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:32.026559    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:33.333247 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:33.333211    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:35.562976 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:35.562943    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:38.084850 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:38.084808    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk" podUID="461e65fc-44a3-437f-93a9-bccc4a4d45b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:42:40.235319 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:40.235270    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:40.721681 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:40.721636    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:41.026203 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:41.026167    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:41.038725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:41.038698    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:41.741127 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:41.741096    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:42:43.101598 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:43.101562    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:43.348465 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:43.348420    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:44.093610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.093582    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"]
Oct 31 19:42:44.097549 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.097531    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.155045 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.155018    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-6279f5-openshift-default-dockercfg-d8cq6\""
Oct 31 19:42:44.179497 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.177844    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.179497 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.177894    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.179497 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.177943    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.179497 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.178030    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg4cs\" (UniqueName: \"kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-kube-api-access-xg4cs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.179497 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.178060    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.182211 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.182174    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2175561-bd09-4247-9c60-772aaf55b972-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.182373 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.182236    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2175561-bd09-4247-9c60-772aaf55b972-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.182373 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.182305    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.182373 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.182332    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.208015 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.207988    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"]
Oct 31 19:42:44.283187 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.283159    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.283306 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.283195    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.283306 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.283236    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.283400 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.283312    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg4cs\" (UniqueName: \"kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-kube-api-access-xg4cs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.283400 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.283339    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.283541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.283393    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2175561-bd09-4247-9c60-772aaf55b972-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.283541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.283431    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2175561-bd09-4247-9c60-772aaf55b972-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.283541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.283528    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.283701 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.283557    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.283701 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.283663    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-certs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.283975 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.283948    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.284267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.284243    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-data\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.284513 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.284467    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-credential-socket\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.284792 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.284763    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2175561-bd09-4247-9c60-772aaf55b972-istiod-ca-cert\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.286052 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.286024    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-envoy\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.287146 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.287118    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2175561-bd09-4247-9c60-772aaf55b972-istio-podinfo\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.344179 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.344111    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-istio-token\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.344290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.344223    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg4cs\" (UniqueName: \"kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-kube-api-access-xg4cs\") pod \"e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") " pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.411457 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.411423    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:44.524003 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.523973    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"
Oct 31 19:42:44.587596 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.587569    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/461e65fc-44a3-437f-93a9-bccc4a4d45b4-kube-api-access-xnh2l\" (UniqueName: \"kubernetes.io/projected/461e65fc-44a3-437f-93a9-bccc4a4d45b4-kube-api-access-xnh2l\") pod \"461e65fc-44a3-437f-93a9-bccc4a4d45b4\" (UID: \"461e65fc-44a3-437f-93a9-bccc4a4d45b4\") "
Oct 31 19:42:44.587777 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.587682    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/461e65fc-44a3-437f-93a9-bccc4a4d45b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461e65fc-44a3-437f-93a9-bccc4a4d45b4-plugins-config-volume\") pod \"461e65fc-44a3-437f-93a9-bccc4a4d45b4\" (UID: \"461e65fc-44a3-437f-93a9-bccc4a4d45b4\") "
Oct 31 19:42:44.588093 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.588070    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/461e65fc-44a3-437f-93a9-bccc4a4d45b4-plugins-config-volume" pod "461e65fc-44a3-437f-93a9-bccc4a4d45b4" (UID: "461e65fc-44a3-437f-93a9-bccc4a4d45b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:44.588174 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.588143    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/461e65fc-44a3-437f-93a9-bccc4a4d45b4-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:44.589870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.589844    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/461e65fc-44a3-437f-93a9-bccc4a4d45b4-kube-api-access-xnh2l" pod "461e65fc-44a3-437f-93a9-bccc4a4d45b4" (UID: "461e65fc-44a3-437f-93a9-bccc4a4d45b4"). InnerVolumeSpecName "kube-api-access-xnh2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:44.689728 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.689657    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnh2l\" (UniqueName: \"kubernetes.io/projected/461e65fc-44a3-437f-93a9-bccc4a4d45b4-kube-api-access-xnh2l\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:42:44.747074 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.747045    4484 generic.go:356] "Generic (PLEG): container finished" podID="461e65fc-44a3-437f-93a9-bccc4a4d45b4" containerID="ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc" exitCode=0
Oct 31 19:42:44.747074 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.747072    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"
Oct 31 19:42:44.747276 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.747109    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk" event={"ID":"461e65fc-44a3-437f-93a9-bccc4a4d45b4","Type":"ContainerDied","Data":"ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc"}
Oct 31 19:42:44.747276 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.747183    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk" event={"ID":"461e65fc-44a3-437f-93a9-bccc4a4d45b4","Type":"ContainerDied","Data":"34d1d8d79017d9de35ab3cb7bf4ec615d11141a4226ed752ef6c1eb0ffab07f6"}
Oct 31 19:42:44.747276 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.747211    4484 scope.go:122] "RemoveContainer" containerID="ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc"
Oct 31 19:42:44.757150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.757135    4484 scope.go:122] "RemoveContainer" containerID="ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc"
Oct 31 19:42:44.757415 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:42:44.757392    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc\": container with ID starting with ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc not found: ID does not exist" containerID="ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc"
Oct 31 19:42:44.757508 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.757424    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc"} err="failed to get container status \"ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc\": rpc error: code = NotFound desc = could not find container \"ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc\": container with ID starting with ecf8b5f2c99e425d2f7c9e68e355a90bedb3add0ea5cb68abe825e45f40ea7cc not found: ID does not exist"
Oct 31 19:42:44.850706 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.850678    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"]
Oct 31 19:42:44.852337 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:42:44.852307    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2175561_bd09_4247_9c60_772aaf55b972.slice/crio-02aa337d4db8fbfb049d12d2bf757e9d45f451bae561fb92fb40ae37a4e8f035 WatchSource:0}: Error finding container 02aa337d4db8fbfb049d12d2bf757e9d45f451bae561fb92fb40ae37a4e8f035: Status 404 returned error can't find the container with id 02aa337d4db8fbfb049d12d2bf757e9d45f451bae561fb92fb40ae37a4e8f035
Oct 31 19:42:44.854665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.854634    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:42:44.854732 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.854697    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:42:44.854732 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.854727    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:42:44.922221 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.922198    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"]
Oct 31 19:42:44.931731 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:44.931703    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-6279f5-568bc68bbf-67qlk"]
Oct 31 19:42:45.139349 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:45.139310    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:45.428064 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:45.427981    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:45.753715 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:45.753675    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5" event={"ID":"a2175561-bd09-4247-9c60-772aaf55b972","Type":"ContainerStarted","Data":"70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d"}
Oct 31 19:42:45.753933 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:45.753736    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5" event={"ID":"a2175561-bd09-4247-9c60-772aaf55b972","Type":"ContainerStarted","Data":"02aa337d4db8fbfb049d12d2bf757e9d45f451bae561fb92fb40ae37a4e8f035"}
Oct 31 19:42:45.885508 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:45.885064    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5" podStartSLOduration=1.8850421910000001 podStartE2EDuration="1.885042191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:45.884596128 +0000 UTC m=+4912.031045994" watchObservedRunningTime="2026-10-31 19:42:45.885042191 +0000 UTC m=+4912.031492056"
Oct 31 19:42:45.995484 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:45.995438    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="461e65fc-44a3-437f-93a9-bccc4a4d45b4" path="/var/lib/kubelet/pods/461e65fc-44a3-437f-93a9-bccc4a4d45b4/volumes"
Oct 31 19:42:46.412091 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:46.412056    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:46.413763 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:46.413741    4484 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 31 19:42:46.413873 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:46.413792    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5" podUID="a2175561-bd09-4247-9c60-772aaf55b972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 31 19:42:47.412174 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:47.412135    4484 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 31 19:42:47.412700 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:47.412193    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5" podUID="a2175561-bd09-4247-9c60-772aaf55b972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 31 19:42:48.412057 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:48.412021    4484 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 31 19:42:48.412249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:48.412075    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5" podUID="a2175561-bd09-4247-9c60-772aaf55b972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 31 19:42:49.345342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:49.345281    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:49.412526 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:49.412459    4484 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 31 19:42:49.412918 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:49.412560    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5" podUID="a2175561-bd09-4247-9c60-772aaf55b972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 31 19:42:50.412227 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:50.412193    4484 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body=
Oct 31 19:42:50.412427 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:50.412261    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5" podUID="a2175561-bd09-4247-9c60-772aaf55b972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused"
Oct 31 19:42:50.879338 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:50.879297    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:51.178651 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:51.178562    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:52.453620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:52.453583    4484 patch_prober.go:29] interesting pod/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:52.454063 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:52.453640    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5" podUID="a2175561-bd09-4247-9c60-772aaf55b972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:53.078809 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:53.078780    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:53.079207 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:53.079179    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:53.080044 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:53.080024    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:42:55.467660 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:42:55.467615    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:42:56.054087 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:56.054052    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:56.634193 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:56.634156    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:57.084217 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:57.084181    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:57.417818 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:57.417728    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:58.797852 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:58.797820    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:59.317530 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:59.317493    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:59.338071 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:59.338040    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"]
Oct 31 19:42:59.338260 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:42:59.338241    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5" podUID="a2175561-bd09-4247-9c60-772aaf55b972" containerName="istio-proxy" containerID="cri-o://70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d" gracePeriod=30
Oct 31 19:43:01.113802 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:01.113760    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:04.394916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.394887    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:04.581273 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.581246    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"]
Oct 31 19:43:04.596481 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.596448    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:43:04.701658 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.701589    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2175561-bd09-4247-9c60-772aaf55b972-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2175561-bd09-4247-9c60-772aaf55b972-istiod-ca-cert\") pod \"a2175561-bd09-4247-9c60-772aaf55b972\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") "
Oct 31 19:43:04.701658 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.701642    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-socket\") pod \"a2175561-bd09-4247-9c60-772aaf55b972\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") "
Oct 31 19:43:04.701855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.701798    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-certs\") pod \"a2175561-bd09-4247-9c60-772aaf55b972\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") "
Oct 31 19:43:04.701923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.701881    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-data\") pod \"a2175561-bd09-4247-9c60-772aaf55b972\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") "
Oct 31 19:43:04.701923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.701892    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-socket" pod "a2175561-bd09-4247-9c60-772aaf55b972" (UID: "a2175561-bd09-4247-9c60-772aaf55b972"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:04.701923 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.701917    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-kube-api-access-xg4cs\" (UniqueName: \"kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-kube-api-access-xg4cs\") pod \"a2175561-bd09-4247-9c60-772aaf55b972\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") "
Oct 31 19:43:04.702095 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.701960    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-istio-token\" (UniqueName: \"kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-istio-token\") pod \"a2175561-bd09-4247-9c60-772aaf55b972\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") "
Oct 31 19:43:04.702095 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.701992    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-envoy\") pod \"a2175561-bd09-4247-9c60-772aaf55b972\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") "
Oct 31 19:43:04.702095 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.702051    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2175561-bd09-4247-9c60-772aaf55b972-istiod-ca-cert" pod "a2175561-bd09-4247-9c60-772aaf55b972" (UID: "a2175561-bd09-4247-9c60-772aaf55b972"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:04.702266 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.702113    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-data" pod "a2175561-bd09-4247-9c60-772aaf55b972" (UID: "a2175561-bd09-4247-9c60-772aaf55b972"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:04.702266 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.702080    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-credential-socket\") pod \"a2175561-bd09-4247-9c60-772aaf55b972\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") "
Oct 31 19:43:04.702266 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.702058    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-certs" pod "a2175561-bd09-4247-9c60-772aaf55b972" (UID: "a2175561-bd09-4247-9c60-772aaf55b972"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:04.702266 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.702167    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2175561-bd09-4247-9c60-772aaf55b972-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2175561-bd09-4247-9c60-772aaf55b972-istio-podinfo\") pod \"a2175561-bd09-4247-9c60-772aaf55b972\" (UID: \"a2175561-bd09-4247-9c60-772aaf55b972\") "
Oct 31 19:43:04.702488 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.702274    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-credential-socket" pod "a2175561-bd09-4247-9c60-772aaf55b972" (UID: "a2175561-bd09-4247-9c60-772aaf55b972"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:04.702588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.702570    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.702652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.702589    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2175561-bd09-4247-9c60-772aaf55b972-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.702652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.702599    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.702652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.702609    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.702652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.702618    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.704346 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.704321    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-istio-token" pod "a2175561-bd09-4247-9c60-772aaf55b972" (UID: "a2175561-bd09-4247-9c60-772aaf55b972"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:04.704522 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.704345    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2175561-bd09-4247-9c60-772aaf55b972-istio-podinfo" pod "a2175561-bd09-4247-9c60-772aaf55b972" (UID: "a2175561-bd09-4247-9c60-772aaf55b972"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:43:04.704522 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.704357    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-envoy" pod "a2175561-bd09-4247-9c60-772aaf55b972" (UID: "a2175561-bd09-4247-9c60-772aaf55b972"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:04.705568 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.705551    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-kube-api-access-xg4cs" pod "a2175561-bd09-4247-9c60-772aaf55b972" (UID: "a2175561-bd09-4247-9c60-772aaf55b972"). InnerVolumeSpecName "kube-api-access-xg4cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:04.804404 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.804369    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2175561-bd09-4247-9c60-772aaf55b972-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.804404 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.804406    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg4cs\" (UniqueName: \"kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-kube-api-access-xg4cs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.804698 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.804423    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2175561-bd09-4247-9c60-772aaf55b972-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.804698 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.804436    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2175561-bd09-4247-9c60-772aaf55b972-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:04.878081 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.877990    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:04.890268 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.890231    4484 generic.go:356] "Generic (PLEG): container finished" podID="a2175561-bd09-4247-9c60-772aaf55b972" containerID="70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d" exitCode=0
Oct 31 19:43:04.890518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.890306    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"
Oct 31 19:43:04.890518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.890378    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5" event={"ID":"a2175561-bd09-4247-9c60-772aaf55b972","Type":"ContainerDied","Data":"70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d"}
Oct 31 19:43:04.890518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.890428    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5" event={"ID":"a2175561-bd09-4247-9c60-772aaf55b972","Type":"ContainerDied","Data":"02aa337d4db8fbfb049d12d2bf757e9d45f451bae561fb92fb40ae37a4e8f035"}
Oct 31 19:43:04.890518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.890457    4484 scope.go:122] "RemoveContainer" containerID="70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d"
Oct 31 19:43:04.903968 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.903944    4484 scope.go:122] "RemoveContainer" containerID="70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d"
Oct 31 19:43:04.904246 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:43:04.904227    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d\": container with ID starting with 70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d not found: ID does not exist" containerID="70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d"
Oct 31 19:43:04.904328 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:04.904254    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d"} err="failed to get container status \"70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d\": rpc error: code = NotFound desc = could not find container \"70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d\": container with ID starting with 70ad99e693a34ce9a977fdf36142c19f8e42792ddf069c8085b19ac052291f1d not found: ID does not exist"
Oct 31 19:43:05.071660 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:05.071628    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"]
Oct 31 19:43:05.078122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:05.078098    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-6279f5-openshift-default-65546c8d99-rwbc5"]
Oct 31 19:43:05.286027 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:05.285998    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:05.993064 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:05.993032    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2175561-bd09-4247-9c60-772aaf55b972" path="/var/lib/kubelet/pods/a2175561-bd09-4247-9c60-772aaf55b972/volumes"
Oct 31 19:43:06.411539 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:06.411448    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:07.930665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:07.930628    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:09.582216 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:09.582175    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl" podUID="30e06783-c3be-4a94-99d1-a60f996d6044" containerName="payload-pre-processing" containerID="cri-o://b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998" gracePeriod=25
Oct 31 19:43:09.846776 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:09.846681    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:10.562936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.562905    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"]
Oct 31 19:43:10.563129 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.563109    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28" podUID="f4594ae3-ae84-47f7-93ca-2295d092f97c" containerName="maas-api" containerID="cri-o://04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59" gracePeriod=30
Oct 31 19:43:10.650309 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.650276    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:10.833750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.833722    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:43:10.932961 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.932929    4484 generic.go:356] "Generic (PLEG): container finished" podID="f4594ae3-ae84-47f7-93ca-2295d092f97c" containerID="04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59" exitCode=0
Oct 31 19:43:10.933158 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.933000    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"
Oct 31 19:43:10.933158 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.932991    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28" event={"ID":"f4594ae3-ae84-47f7-93ca-2295d092f97c","Type":"ContainerDied","Data":"04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59"}
Oct 31 19:43:10.933264 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.933161    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28" event={"ID":"f4594ae3-ae84-47f7-93ca-2295d092f97c","Type":"ContainerDied","Data":"fc0cd3778f4f955abc92168aeaffffe22c4b2f016f960c9b649bf8dc965ed9d6"}
Oct 31 19:43:10.933264 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.933186    4484 scope.go:122] "RemoveContainer" containerID="04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59"
Oct 31 19:43:10.942665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.942643    4484 scope.go:122] "RemoveContainer" containerID="04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59"
Oct 31 19:43:10.942967 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:43:10.942945    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59\": container with ID starting with 04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59 not found: ID does not exist" containerID="04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59"
Oct 31 19:43:10.943070 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.942972    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59"} err="failed to get container status \"04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59\": rpc error: code = NotFound desc = could not find container \"04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59\": container with ID starting with 04135f94a27d88d1581ed1d44813f2c290875aa34580c5ab233384a4344c3e59 not found: ID does not exist"
Oct 31 19:43:10.969183 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.969148    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4594ae3-ae84-47f7-93ca-2295d092f97c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4594ae3-ae84-47f7-93ca-2295d092f97c-openshift-service-ca\") pod \"f4594ae3-ae84-47f7-93ca-2295d092f97c\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") "
Oct 31 19:43:10.969389 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.969200    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4594ae3-ae84-47f7-93ca-2295d092f97c-kube-api-access-p5zjc\" (UniqueName: \"kubernetes.io/projected/f4594ae3-ae84-47f7-93ca-2295d092f97c-kube-api-access-p5zjc\") pod \"f4594ae3-ae84-47f7-93ca-2295d092f97c\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") "
Oct 31 19:43:10.969389 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.969241    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-metrics-certs\") pod \"f4594ae3-ae84-47f7-93ca-2295d092f97c\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") "
Oct 31 19:43:10.969542 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.969414    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-maas-api-tls\") pod \"f4594ae3-ae84-47f7-93ca-2295d092f97c\" (UID: \"f4594ae3-ae84-47f7-93ca-2295d092f97c\") "
Oct 31 19:43:10.969639 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.969610    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4594ae3-ae84-47f7-93ca-2295d092f97c-openshift-service-ca" pod "f4594ae3-ae84-47f7-93ca-2295d092f97c" (UID: "f4594ae3-ae84-47f7-93ca-2295d092f97c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:10.969927 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.969905    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4594ae3-ae84-47f7-93ca-2295d092f97c-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:10.971730 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.971700    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4594ae3-ae84-47f7-93ca-2295d092f97c-kube-api-access-p5zjc" pod "f4594ae3-ae84-47f7-93ca-2295d092f97c" (UID: "f4594ae3-ae84-47f7-93ca-2295d092f97c"). InnerVolumeSpecName "kube-api-access-p5zjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:10.972199 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.972180    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-metrics-certs" pod "f4594ae3-ae84-47f7-93ca-2295d092f97c" (UID: "f4594ae3-ae84-47f7-93ca-2295d092f97c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:10.972262 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:10.972201    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-maas-api-tls" pod "f4594ae3-ae84-47f7-93ca-2295d092f97c" (UID: "f4594ae3-ae84-47f7-93ca-2295d092f97c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:11.070936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:11.070847    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:11.070936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:11.070877    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5zjc\" (UniqueName: \"kubernetes.io/projected/f4594ae3-ae84-47f7-93ca-2295d092f97c-kube-api-access-p5zjc\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:11.070936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:11.070890    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4594ae3-ae84-47f7-93ca-2295d092f97c-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:11.355421 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:11.355337    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"]
Oct 31 19:43:11.406230 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:11.406191    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-dbceb9-76f58ffd68-gtm28"]
Oct 31 19:43:11.992711 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:11.992682    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4594ae3-ae84-47f7-93ca-2295d092f97c" path="/var/lib/kubelet/pods/f4594ae3-ae84-47f7-93ca-2295d092f97c/volumes"
Oct 31 19:43:12.446613 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:12.446521    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:12.702816 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:12.702738    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:15.903141 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:15.903111    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:16.205084 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:16.204996    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:16.436078 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:16.436040    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:43:17.450868 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:17.450835    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:17.728131 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:17.728038    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:18.002059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:18.002018    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl" podUID="30e06783-c3be-4a94-99d1-a60f996d6044" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:43:18.108145 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:18.108104    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:18.369552 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:18.369436    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:18.896870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:18.896838    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:22.638963 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:22.638924    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:25.381632 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:25.381592    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"]
Oct 31 19:43:25.469490 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:43:25.469442    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:43:25.610163 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:25.610123    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:25.963352 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:25.963290    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:27.650580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:27.650530    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:28.002341 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:28.002301    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl" podUID="30e06783-c3be-4a94-99d1-a60f996d6044" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:43:29.845520 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:29.845450    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"
Oct 31 19:43:29.960942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:29.960911    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e06783-c3be-4a94-99d1-a60f996d6044-kube-api-access-8hdsz\" (UniqueName: \"kubernetes.io/projected/30e06783-c3be-4a94-99d1-a60f996d6044-kube-api-access-8hdsz\") pod \"30e06783-c3be-4a94-99d1-a60f996d6044\" (UID: \"30e06783-c3be-4a94-99d1-a60f996d6044\") "
Oct 31 19:43:29.961141 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:29.961022    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e06783-c3be-4a94-99d1-a60f996d6044-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e06783-c3be-4a94-99d1-a60f996d6044-plugins-config-volume\") pod \"30e06783-c3be-4a94-99d1-a60f996d6044\" (UID: \"30e06783-c3be-4a94-99d1-a60f996d6044\") "
Oct 31 19:43:29.961387 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:29.961363    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e06783-c3be-4a94-99d1-a60f996d6044-plugins-config-volume" pod "30e06783-c3be-4a94-99d1-a60f996d6044" (UID: "30e06783-c3be-4a94-99d1-a60f996d6044"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:29.963200 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:29.963183    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e06783-c3be-4a94-99d1-a60f996d6044-kube-api-access-8hdsz" pod "30e06783-c3be-4a94-99d1-a60f996d6044" (UID: "30e06783-c3be-4a94-99d1-a60f996d6044"). InnerVolumeSpecName "kube-api-access-8hdsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:30.035412 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:30.035384    4484 generic.go:356] "Generic (PLEG): container finished" podID="30e06783-c3be-4a94-99d1-a60f996d6044" containerID="b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998" exitCode=0
Oct 31 19:43:30.035592 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:30.035432    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"
Oct 31 19:43:30.035592 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:30.035468    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl" event={"ID":"30e06783-c3be-4a94-99d1-a60f996d6044","Type":"ContainerDied","Data":"b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998"}
Oct 31 19:43:30.035592 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:30.035541    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl" event={"ID":"30e06783-c3be-4a94-99d1-a60f996d6044","Type":"ContainerDied","Data":"57dd89a6be68bfc721463966568847847576f66e3c5ceb3e88055213f0e05e14"}
Oct 31 19:43:30.035592 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:30.035564    4484 scope.go:122] "RemoveContainer" containerID="b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998"
Oct 31 19:43:30.045719 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:30.045625    4484 scope.go:122] "RemoveContainer" containerID="b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998"
Oct 31 19:43:30.046868 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:43:30.046840    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998\": container with ID starting with b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998 not found: ID does not exist" containerID="b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998"
Oct 31 19:43:30.047029 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:30.046879    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998"} err="failed to get container status \"b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998\": rpc error: code = NotFound desc = could not find container \"b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998\": container with ID starting with b99ed90a79c6ba9f4642c20d914de517e1b2c2a57b7eedbab9fa68ae0e2e7998 not found: ID does not exist"
Oct 31 19:43:30.062614 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:30.062594    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hdsz\" (UniqueName: \"kubernetes.io/projected/30e06783-c3be-4a94-99d1-a60f996d6044-kube-api-access-8hdsz\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:30.062614 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:30.062615    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e06783-c3be-4a94-99d1-a60f996d6044-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:30.221388 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:30.221292    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"]
Oct 31 19:43:30.225415 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:30.225391    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dbceb9-5685c65f5f-v8tbl"]
Oct 31 19:43:30.382824 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:30.382789    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc" podUID="b0cc9f9a-c164-4c68-a62c-9b8832aa0850" containerName="payload-pre-processing" containerID="cri-o://b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d" gracePeriod=25
Oct 31 19:43:31.417164 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:31.417113    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:31.992675 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:31.992638    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e06783-c3be-4a94-99d1-a60f996d6044" path="/var/lib/kubelet/pods/30e06783-c3be-4a94-99d1-a60f996d6044/volumes"
Oct 31 19:43:32.637353 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:32.637303    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:35.457149 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:35.457112    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:35.818538 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:35.818463    4484 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:35.834295 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:35.834266    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc" podUID="b0cc9f9a-c164-4c68-a62c-9b8832aa0850" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:43:45.834333 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:45.834295    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc" podUID="b0cc9f9a-c164-4c68-a62c-9b8832aa0850" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:43:46.486493 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.486448    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"]
Oct 31 19:43:46.486690 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.486669    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw" podUID="41ff7b73-ecc2-4890-9ae0-284e55764162" containerName="maas-api" containerID="cri-o://772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5" gracePeriod=30
Oct 31 19:43:46.749710 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.749649    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:43:46.836095 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.836060    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-maas-api-tls\") pod \"41ff7b73-ecc2-4890-9ae0-284e55764162\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") "
Oct 31 19:43:46.836540 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.836143    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ff7b73-ecc2-4890-9ae0-284e55764162-kube-api-access-snqq4\" (UniqueName: \"kubernetes.io/projected/41ff7b73-ecc2-4890-9ae0-284e55764162-kube-api-access-snqq4\") pod \"41ff7b73-ecc2-4890-9ae0-284e55764162\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") "
Oct 31 19:43:46.836540 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.836176    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-metrics-certs\") pod \"41ff7b73-ecc2-4890-9ae0-284e55764162\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") "
Oct 31 19:43:46.838389 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.838361    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-maas-api-tls" pod "41ff7b73-ecc2-4890-9ae0-284e55764162" (UID: "41ff7b73-ecc2-4890-9ae0-284e55764162"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:46.838519 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.838390    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-metrics-certs" pod "41ff7b73-ecc2-4890-9ae0-284e55764162" (UID: "41ff7b73-ecc2-4890-9ae0-284e55764162"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:46.838519 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.838399    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ff7b73-ecc2-4890-9ae0-284e55764162-kube-api-access-snqq4" pod "41ff7b73-ecc2-4890-9ae0-284e55764162" (UID: "41ff7b73-ecc2-4890-9ae0-284e55764162"). InnerVolumeSpecName "kube-api-access-snqq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:46.937695 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.937662    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ff7b73-ecc2-4890-9ae0-284e55764162-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ff7b73-ecc2-4890-9ae0-284e55764162-openshift-service-ca\") pod \"41ff7b73-ecc2-4890-9ae0-284e55764162\" (UID: \"41ff7b73-ecc2-4890-9ae0-284e55764162\") "
Oct 31 19:43:46.937973 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.937955    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:46.938047 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.937979    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snqq4\" (UniqueName: \"kubernetes.io/projected/41ff7b73-ecc2-4890-9ae0-284e55764162-kube-api-access-snqq4\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:46.938047 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.937993    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ff7b73-ecc2-4890-9ae0-284e55764162-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:46.938047 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:46.938001    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ff7b73-ecc2-4890-9ae0-284e55764162-openshift-service-ca" pod "41ff7b73-ecc2-4890-9ae0-284e55764162" (UID: "41ff7b73-ecc2-4890-9ae0-284e55764162"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:47.039583 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:47.039503    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ff7b73-ecc2-4890-9ae0-284e55764162-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:47.124365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:47.124336    4484 generic.go:356] "Generic (PLEG): container finished" podID="41ff7b73-ecc2-4890-9ae0-284e55764162" containerID="772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5" exitCode=0
Oct 31 19:43:47.124567 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:47.124377    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"
Oct 31 19:43:47.124567 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:47.124432    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw" event={"ID":"41ff7b73-ecc2-4890-9ae0-284e55764162","Type":"ContainerDied","Data":"772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5"}
Oct 31 19:43:47.124567 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:47.124497    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw" event={"ID":"41ff7b73-ecc2-4890-9ae0-284e55764162","Type":"ContainerDied","Data":"3807ab3a2a283b472df923099d9e9c2d26525897f99d60b6bc52151364fc06c7"}
Oct 31 19:43:47.124567 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:47.124523    4484 scope.go:122] "RemoveContainer" containerID="772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5"
Oct 31 19:43:47.134722 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:47.134706    4484 scope.go:122] "RemoveContainer" containerID="772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5"
Oct 31 19:43:47.134976 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:43:47.134958    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5\": container with ID starting with 772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5 not found: ID does not exist" containerID="772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5"
Oct 31 19:43:47.135034 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:47.134989    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5"} err="failed to get container status \"772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5\": rpc error: code = NotFound desc = could not find container \"772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5\": container with ID starting with 772f9e8677fe2eb075e46f8d4cf4bcb60f81cd247b93e81bbe39a719c48da8a5 not found: ID does not exist"
Oct 31 19:43:47.311337 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:47.311250    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"]
Oct 31 19:43:47.313647 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:47.313625    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6cfd68-69b7ccb9c-4lxvw"]
Oct 31 19:43:47.993318 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:47.993283    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ff7b73-ecc2-4890-9ae0-284e55764162" path="/var/lib/kubelet/pods/41ff7b73-ecc2-4890-9ae0-284e55764162/volumes"
Oct 31 19:43:50.242350 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:50.242310    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"]
Oct 31 19:43:50.242849 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:50.242582    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh" podUID="d0fc3a9e-624f-4971-97e8-4a4e3ff08181" containerName="istio-proxy" containerID="cri-o://87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa" gracePeriod=30
Oct 31 19:43:50.638424 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:50.638397    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"
Oct 31 19:43:50.671011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:50.670981    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-plugins-config-volume\") pod \"b0cc9f9a-c164-4c68-a62c-9b8832aa0850\" (UID: \"b0cc9f9a-c164-4c68-a62c-9b8832aa0850\") "
Oct 31 19:43:50.671172 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:50.671027    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-kube-api-access-97qt8\" (UniqueName: \"kubernetes.io/projected/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-kube-api-access-97qt8\") pod \"b0cc9f9a-c164-4c68-a62c-9b8832aa0850\" (UID: \"b0cc9f9a-c164-4c68-a62c-9b8832aa0850\") "
Oct 31 19:43:50.671379 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:50.671356    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-plugins-config-volume" pod "b0cc9f9a-c164-4c68-a62c-9b8832aa0850" (UID: "b0cc9f9a-c164-4c68-a62c-9b8832aa0850"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:50.671520 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:50.671500    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:50.674126 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:50.674100    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-kube-api-access-97qt8" pod "b0cc9f9a-c164-4c68-a62c-9b8832aa0850" (UID: "b0cc9f9a-c164-4c68-a62c-9b8832aa0850"). InnerVolumeSpecName "kube-api-access-97qt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:50.772179 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:50.772115    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97qt8\" (UniqueName: \"kubernetes.io/projected/b0cc9f9a-c164-4c68-a62c-9b8832aa0850-kube-api-access-97qt8\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:51.148947 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.148857    4484 generic.go:356] "Generic (PLEG): container finished" podID="b0cc9f9a-c164-4c68-a62c-9b8832aa0850" containerID="b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d" exitCode=0
Oct 31 19:43:51.148947 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.148901    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"
Oct 31 19:43:51.148947 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.148911    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc" event={"ID":"b0cc9f9a-c164-4c68-a62c-9b8832aa0850","Type":"ContainerDied","Data":"b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d"}
Oct 31 19:43:51.149297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.148959    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc" event={"ID":"b0cc9f9a-c164-4c68-a62c-9b8832aa0850","Type":"ContainerDied","Data":"ed6a6ebdbed8feaf1361e677c4c895875a04c1e89ec05cf794b2b68757544b52"}
Oct 31 19:43:51.149297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.148987    4484 scope.go:122] "RemoveContainer" containerID="b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d"
Oct 31 19:43:51.158676 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.158660    4484 scope.go:122] "RemoveContainer" containerID="b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d"
Oct 31 19:43:51.158944 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:43:51.158923    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d\": container with ID starting with b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d not found: ID does not exist" containerID="b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d"
Oct 31 19:43:51.158998 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.158952    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d"} err="failed to get container status \"b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d\": rpc error: code = NotFound desc = could not find container \"b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d\": container with ID starting with b7fbc9b6d13fbb0f21a0a46e26315b7e703a4f94c327f5bba22934eb087cfe8d not found: ID does not exist"
Oct 31 19:43:51.317024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.316992    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"]
Oct 31 19:43:51.367939 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.367907    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6cfd68-789647ccc5-q7gvc"]
Oct 31 19:43:51.393281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393256    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"]
Oct 31 19:43:51.393713 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393700    4484 cpu_manager.go:414] "containerMap: removing container" podUID="461e65fc-44a3-437f-93a9-bccc4a4d45b4" containerName="payload-processing"
Oct 31 19:43:51.393764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393715    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="461e65fc-44a3-437f-93a9-bccc4a4d45b4" containerName="payload-processing"
Oct 31 19:43:51.393764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393728    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f4594ae3-ae84-47f7-93ca-2295d092f97c" containerName="maas-api"
Oct 31 19:43:51.393764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393733    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4594ae3-ae84-47f7-93ca-2295d092f97c" containerName="maas-api"
Oct 31 19:43:51.393764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393750    4484 cpu_manager.go:414] "containerMap: removing container" podUID="41ff7b73-ecc2-4890-9ae0-284e55764162" containerName="maas-api"
Oct 31 19:43:51.393764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393756    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ff7b73-ecc2-4890-9ae0-284e55764162" containerName="maas-api"
Oct 31 19:43:51.393926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393767    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a2175561-bd09-4247-9c60-772aaf55b972" containerName="istio-proxy"
Oct 31 19:43:51.393926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393772    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2175561-bd09-4247-9c60-772aaf55b972" containerName="istio-proxy"
Oct 31 19:43:51.393926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393779    4484 cpu_manager.go:414] "containerMap: removing container" podUID="b0cc9f9a-c164-4c68-a62c-9b8832aa0850" containerName="payload-pre-processing"
Oct 31 19:43:51.393926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393784    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cc9f9a-c164-4c68-a62c-9b8832aa0850" containerName="payload-pre-processing"
Oct 31 19:43:51.393926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393792    4484 cpu_manager.go:414] "containerMap: removing container" podUID="30e06783-c3be-4a94-99d1-a60f996d6044" containerName="payload-pre-processing"
Oct 31 19:43:51.393926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393797    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e06783-c3be-4a94-99d1-a60f996d6044" containerName="payload-pre-processing"
Oct 31 19:43:51.393926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393874    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="461e65fc-44a3-437f-93a9-bccc4a4d45b4" containerName="payload-processing"
Oct 31 19:43:51.393926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393882    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4594ae3-ae84-47f7-93ca-2295d092f97c" containerName="maas-api"
Oct 31 19:43:51.393926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393890    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ff7b73-ecc2-4890-9ae0-284e55764162" containerName="maas-api"
Oct 31 19:43:51.393926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393897    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2175561-bd09-4247-9c60-772aaf55b972" containerName="istio-proxy"
Oct 31 19:43:51.393926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393902    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cc9f9a-c164-4c68-a62c-9b8832aa0850" containerName="payload-pre-processing"
Oct 31 19:43:51.393926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.393908    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e06783-c3be-4a94-99d1-a60f996d6044" containerName="payload-pre-processing"
Oct 31 19:43:51.398284 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.398269    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:51.452918 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.452830    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:43:51.453269 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.453246    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:43:51.453774 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.453747    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"]
Oct 31 19:43:51.477899 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.477865    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtgts\" (UniqueName: \"kubernetes.io/projected/a67bd72e-4248-47c8-9b99-7e2393bea074-kube-api-access-mtgts\") pod \"maas-api-revoke-sub-acafd54d-xh4p6\" (UID: \"a67bd72e-4248-47c8-9b99-7e2393bea074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:51.478043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.477913    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acafd54d-xh4p6\" (UID: \"a67bd72e-4248-47c8-9b99-7e2393bea074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:51.478043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.477940    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acafd54d-xh4p6\" (UID: \"a67bd72e-4248-47c8-9b99-7e2393bea074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:51.578590 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.578558    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgts\" (UniqueName: \"kubernetes.io/projected/a67bd72e-4248-47c8-9b99-7e2393bea074-kube-api-access-mtgts\") pod \"maas-api-revoke-sub-acafd54d-xh4p6\" (UID: \"a67bd72e-4248-47c8-9b99-7e2393bea074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:51.578804 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.578613    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acafd54d-xh4p6\" (UID: \"a67bd72e-4248-47c8-9b99-7e2393bea074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:51.578804 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.578649    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acafd54d-xh4p6\" (UID: \"a67bd72e-4248-47c8-9b99-7e2393bea074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:51.579382 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.579306    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acafd54d-xh4p6\" (UID: \"a67bd72e-4248-47c8-9b99-7e2393bea074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:51.581060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.581038    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acafd54d-xh4p6\" (UID: \"a67bd72e-4248-47c8-9b99-7e2393bea074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:51.638224 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.638190    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtgts\" (UniqueName: \"kubernetes.io/projected/a67bd72e-4248-47c8-9b99-7e2393bea074-kube-api-access-mtgts\") pod \"maas-api-revoke-sub-acafd54d-xh4p6\" (UID: \"a67bd72e-4248-47c8-9b99-7e2393bea074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:51.711018 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.710933    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:51.996640 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:51.994837    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cc9f9a-c164-4c68-a62c-9b8832aa0850" path="/var/lib/kubelet/pods/b0cc9f9a-c164-4c68-a62c-9b8832aa0850/volumes"
Oct 31 19:43:52.002582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:52.002546    4484 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:43:52.002746 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:52.002618    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh" podUID="d0fc3a9e-624f-4971-97e8-4a4e3ff08181" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:43:52.093019 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:52.092992    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"]
Oct 31 19:43:52.095148 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:43:52.095121    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67bd72e_4248_47c8_9b99_7e2393bea074.slice/crio-4749ed87e7a7ff96f404b5ad2266b68a4f7e98aeca5f1bc3d518af307bcb9e24 WatchSource:0}: Error finding container 4749ed87e7a7ff96f404b5ad2266b68a4f7e98aeca5f1bc3d518af307bcb9e24: Status 404 returned error can't find the container with id 4749ed87e7a7ff96f404b5ad2266b68a4f7e98aeca5f1bc3d518af307bcb9e24
Oct 31 19:43:52.158772 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:52.158739    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6" event={"ID":"a67bd72e-4248-47c8-9b99-7e2393bea074","Type":"ContainerStarted","Data":"4749ed87e7a7ff96f404b5ad2266b68a4f7e98aeca5f1bc3d518af307bcb9e24"}
Oct 31 19:43:53.165650 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:53.165616    4484 generic.go:356] "Generic (PLEG): container finished" podID="a67bd72e-4248-47c8-9b99-7e2393bea074" containerID="9f17774abc28d4cae886197d517cd47322e79ee1f61405fdbc9f84313637ae77" exitCode=0
Oct 31 19:43:53.166099 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:53.165706    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6" event={"ID":"a67bd72e-4248-47c8-9b99-7e2393bea074","Type":"ContainerDied","Data":"9f17774abc28d4cae886197d517cd47322e79ee1f61405fdbc9f84313637ae77"}
Oct 31 19:43:54.308322 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:54.308298    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:54.406202 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:54.406173    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-xks-ca\") pod \"a67bd72e-4248-47c8-9b99-7e2393bea074\" (UID: \"a67bd72e-4248-47c8-9b99-7e2393bea074\") "
Oct 31 19:43:54.406355 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:54.406221    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67bd72e-4248-47c8-9b99-7e2393bea074-kube-api-access-mtgts\" (UniqueName: \"kubernetes.io/projected/a67bd72e-4248-47c8-9b99-7e2393bea074-kube-api-access-mtgts\") pod \"a67bd72e-4248-47c8-9b99-7e2393bea074\" (UID: \"a67bd72e-4248-47c8-9b99-7e2393bea074\") "
Oct 31 19:43:54.406410 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:54.406395    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-openshift-ca\") pod \"a67bd72e-4248-47c8-9b99-7e2393bea074\" (UID: \"a67bd72e-4248-47c8-9b99-7e2393bea074\") "
Oct 31 19:43:54.406730 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:54.406710    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-openshift-ca" pod "a67bd72e-4248-47c8-9b99-7e2393bea074" (UID: "a67bd72e-4248-47c8-9b99-7e2393bea074"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:54.408435 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:54.408411    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-xks-ca" pod "a67bd72e-4248-47c8-9b99-7e2393bea074" (UID: "a67bd72e-4248-47c8-9b99-7e2393bea074"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:54.408435 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:54.408416    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67bd72e-4248-47c8-9b99-7e2393bea074-kube-api-access-mtgts" pod "a67bd72e-4248-47c8-9b99-7e2393bea074" (UID: "a67bd72e-4248-47c8-9b99-7e2393bea074"). InnerVolumeSpecName "kube-api-access-mtgts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:54.507336 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:54.507309    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:54.507336 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:54.507334    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67bd72e-4248-47c8-9b99-7e2393bea074-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:54.507571 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:54.507344    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtgts\" (UniqueName: \"kubernetes.io/projected/a67bd72e-4248-47c8-9b99-7e2393bea074-kube-api-access-mtgts\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.181325 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.181279    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6" event={"ID":"a67bd72e-4248-47c8-9b99-7e2393bea074","Type":"ContainerDied","Data":"4749ed87e7a7ff96f404b5ad2266b68a4f7e98aeca5f1bc3d518af307bcb9e24"}
Oct 31 19:43:55.181560 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.181348    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4749ed87e7a7ff96f404b5ad2266b68a4f7e98aeca5f1bc3d518af307bcb9e24"
Oct 31 19:43:55.181560 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.181414    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"
Oct 31 19:43:55.468537 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:43:55.468392    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:43:55.493717 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.493697    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:43:55.516711 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.516688    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-credential-socket\") pod \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") "
Oct 31 19:43:55.516855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.516727    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-podinfo\") pod \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") "
Oct 31 19:43:55.516855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.516766    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-kube-api-access-h9z5z\" (UniqueName: \"kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-kube-api-access-h9z5z\") pod \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") "
Oct 31 19:43:55.516981 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.516857    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-token\" (UniqueName: \"kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-token\") pod \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") "
Oct 31 19:43:55.516981 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.516908    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-certs\") pod \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") "
Oct 31 19:43:55.516981 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.516940    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-socket\") pod \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") "
Oct 31 19:43:55.517173 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.517026    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-envoy\") pod \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") "
Oct 31 19:43:55.517173 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.517041    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-credential-socket" pod "d0fc3a9e-624f-4971-97e8-4a4e3ff08181" (UID: "d0fc3a9e-624f-4971-97e8-4a4e3ff08181"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:55.517173 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.517073    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-data\") pod \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") "
Oct 31 19:43:55.517173 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.517112    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istiod-ca-cert\") pod \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\" (UID: \"d0fc3a9e-624f-4971-97e8-4a4e3ff08181\") "
Oct 31 19:43:55.517424 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.517204    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-certs" pod "d0fc3a9e-624f-4971-97e8-4a4e3ff08181" (UID: "d0fc3a9e-624f-4971-97e8-4a4e3ff08181"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:55.517506 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.517282    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-socket" pod "d0fc3a9e-624f-4971-97e8-4a4e3ff08181" (UID: "d0fc3a9e-624f-4971-97e8-4a4e3ff08181"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:55.517506 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.517384    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-data" pod "d0fc3a9e-624f-4971-97e8-4a4e3ff08181" (UID: "d0fc3a9e-624f-4971-97e8-4a4e3ff08181"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:55.517657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.517638    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istiod-ca-cert" pod "d0fc3a9e-624f-4971-97e8-4a4e3ff08181" (UID: "d0fc3a9e-624f-4971-97e8-4a4e3ff08181"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:55.517733 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.517716    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.517796 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.517738    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.517796 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.517756    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.517796 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.517771    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.519115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.519094    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-kube-api-access-h9z5z" pod "d0fc3a9e-624f-4971-97e8-4a4e3ff08181" (UID: "d0fc3a9e-624f-4971-97e8-4a4e3ff08181"). InnerVolumeSpecName "kube-api-access-h9z5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:55.519661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.519637    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-token" pod "d0fc3a9e-624f-4971-97e8-4a4e3ff08181" (UID: "d0fc3a9e-624f-4971-97e8-4a4e3ff08181"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:55.519661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.519652    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-podinfo" pod "d0fc3a9e-624f-4971-97e8-4a4e3ff08181" (UID: "d0fc3a9e-624f-4971-97e8-4a4e3ff08181"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:43:55.519842 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.519743    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-envoy" pod "d0fc3a9e-624f-4971-97e8-4a4e3ff08181" (UID: "d0fc3a9e-624f-4971-97e8-4a4e3ff08181"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:55.619041 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.619004    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.619041 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.619041    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9z5z\" (UniqueName: \"kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-kube-api-access-h9z5z\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.619289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.619057    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.619289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.619069    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:55.619289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:55.619083    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0fc3a9e-624f-4971-97e8-4a4e3ff08181-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:43:56.188648 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:56.188561    4484 generic.go:356] "Generic (PLEG): container finished" podID="d0fc3a9e-624f-4971-97e8-4a4e3ff08181" containerID="87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa" exitCode=0
Oct 31 19:43:56.188648 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:56.188617    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"
Oct 31 19:43:56.188648 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:56.188624    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh" event={"ID":"d0fc3a9e-624f-4971-97e8-4a4e3ff08181","Type":"ContainerDied","Data":"87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa"}
Oct 31 19:43:56.188916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:56.188675    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh" event={"ID":"d0fc3a9e-624f-4971-97e8-4a4e3ff08181","Type":"ContainerDied","Data":"e6cd72b81dfe746a99b744a9e9895312bd132189ab6716249114bbff8467af7c"}
Oct 31 19:43:56.188916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:56.188704    4484 scope.go:122] "RemoveContainer" containerID="87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa"
Oct 31 19:43:56.197619 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:56.197597    4484 scope.go:122] "RemoveContainer" containerID="87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa"
Oct 31 19:43:56.197913 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:43:56.197892    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa\": container with ID starting with 87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa not found: ID does not exist" containerID="87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa"
Oct 31 19:43:56.197958 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:56.197922    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa"} err="failed to get container status \"87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa\": rpc error: code = NotFound desc = could not find container \"87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa\": container with ID starting with 87831dfe465d03c0575a9fb6be0848df383b72d08674223326cfa22881cf45aa not found: ID does not exist"
Oct 31 19:43:56.369461 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:56.369428    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"]
Oct 31 19:43:56.371333 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:56.371312    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6cfd68-openshift-default-764ffdc779-j6znh"]
Oct 31 19:43:56.864678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:56.864647    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:43:57.993963 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:43:57.993932    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0fc3a9e-624f-4971-97e8-4a4e3ff08181" path="/var/lib/kubelet/pods/d0fc3a9e-624f-4971-97e8-4a4e3ff08181/volumes"
Oct 31 19:44:04.640133 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.640100    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"]
Oct 31 19:44:04.640571 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.640558    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a67bd72e-4248-47c8-9b99-7e2393bea074" containerName="revoke-keys"
Oct 31 19:44:04.640622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.640573    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67bd72e-4248-47c8-9b99-7e2393bea074" containerName="revoke-keys"
Oct 31 19:44:04.640622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.640598    4484 cpu_manager.go:414] "containerMap: removing container" podUID="d0fc3a9e-624f-4971-97e8-4a4e3ff08181" containerName="istio-proxy"
Oct 31 19:44:04.640622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.640603    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0fc3a9e-624f-4971-97e8-4a4e3ff08181" containerName="istio-proxy"
Oct 31 19:44:04.640734 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.640664    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67bd72e-4248-47c8-9b99-7e2393bea074" containerName="revoke-keys"
Oct 31 19:44:04.640734 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.640676    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0fc3a9e-624f-4971-97e8-4a4e3ff08181" containerName="istio-proxy"
Oct 31 19:44:04.645239 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.645223    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:04.706357 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.706324    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"]
Oct 31 19:44:04.710503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.710456    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccf9fefc-fkg48\" (UID: \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:04.710712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.710692    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw56q\" (UniqueName: \"kubernetes.io/projected/cca0e51a-2c91-4c8e-ac54-e682fe88271a-kube-api-access-vw56q\") pod \"maas-api-revoke-sub-ccf9fefc-fkg48\" (UID: \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:04.710814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.710734    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccf9fefc-fkg48\" (UID: \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:04.754630 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.754598    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:44:04.754798 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.754661    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:44:04.811622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.811594    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw56q\" (UniqueName: \"kubernetes.io/projected/cca0e51a-2c91-4c8e-ac54-e682fe88271a-kube-api-access-vw56q\") pod \"maas-api-revoke-sub-ccf9fefc-fkg48\" (UID: \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:04.811795 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.811637    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccf9fefc-fkg48\" (UID: \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:04.811795 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.811669    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccf9fefc-fkg48\" (UID: \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:04.812340 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.812306    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccf9fefc-fkg48\" (UID: \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:04.814365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.814335    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccf9fefc-fkg48\" (UID: \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:04.876009 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.875981    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw56q\" (UniqueName: \"kubernetes.io/projected/cca0e51a-2c91-4c8e-ac54-e682fe88271a-kube-api-access-vw56q\") pod \"maas-api-revoke-sub-ccf9fefc-fkg48\" (UID: \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:04.957913 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:04.957839    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:05.349299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:05.349268    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"]
Oct 31 19:44:05.349986 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:44:05.349962    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca0e51a_2c91_4c8e_ac54_e682fe88271a.slice/crio-32be5d248ea4fb76b62aa8a30536d97042f586d3ac8d555914df96a51550b503 WatchSource:0}: Error finding container 32be5d248ea4fb76b62aa8a30536d97042f586d3ac8d555914df96a51550b503: Status 404 returned error can't find the container with id 32be5d248ea4fb76b62aa8a30536d97042f586d3ac8d555914df96a51550b503
Oct 31 19:44:06.242553 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:06.242523    4484 generic.go:356] "Generic (PLEG): container finished" podID="cca0e51a-2c91-4c8e-ac54-e682fe88271a" containerID="c6d56b3b9f3d5d7cc18b5e966d946bfbd78c82940cebca1eb3b9e748d7053c4f" exitCode=0
Oct 31 19:44:06.242954 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:06.242607    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48" event={"ID":"cca0e51a-2c91-4c8e-ac54-e682fe88271a","Type":"ContainerDied","Data":"c6d56b3b9f3d5d7cc18b5e966d946bfbd78c82940cebca1eb3b9e748d7053c4f"}
Oct 31 19:44:06.242954 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:06.242665    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48" event={"ID":"cca0e51a-2c91-4c8e-ac54-e682fe88271a","Type":"ContainerStarted","Data":"32be5d248ea4fb76b62aa8a30536d97042f586d3ac8d555914df96a51550b503"}
Oct 31 19:44:06.451885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:06.451849    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"]
Oct 31 19:44:06.543646 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:06.543612    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"]
Oct 31 19:44:07.389186 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:07.389155    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:07.436420 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:07.436388    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-openshift-ca\") pod \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\" (UID: \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\") "
Oct 31 19:44:07.436610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:07.436511    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-xks-ca\") pod \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\" (UID: \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\") "
Oct 31 19:44:07.436610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:07.436557    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca0e51a-2c91-4c8e-ac54-e682fe88271a-kube-api-access-vw56q\" (UniqueName: \"kubernetes.io/projected/cca0e51a-2c91-4c8e-ac54-e682fe88271a-kube-api-access-vw56q\") pod \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\" (UID: \"cca0e51a-2c91-4c8e-ac54-e682fe88271a\") "
Oct 31 19:44:07.436814 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:07.436786    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-openshift-ca" pod "cca0e51a-2c91-4c8e-ac54-e682fe88271a" (UID: "cca0e51a-2c91-4c8e-ac54-e682fe88271a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:07.438697 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:07.438667    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-xks-ca" pod "cca0e51a-2c91-4c8e-ac54-e682fe88271a" (UID: "cca0e51a-2c91-4c8e-ac54-e682fe88271a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:44:07.438807 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:07.438765    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca0e51a-2c91-4c8e-ac54-e682fe88271a-kube-api-access-vw56q" pod "cca0e51a-2c91-4c8e-ac54-e682fe88271a" (UID: "cca0e51a-2c91-4c8e-ac54-e682fe88271a"). InnerVolumeSpecName "kube-api-access-vw56q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:07.537292 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:07.537218    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:07.537292 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:07.537244    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cca0e51a-2c91-4c8e-ac54-e682fe88271a-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:07.537292 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:07.537253    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw56q\" (UniqueName: \"kubernetes.io/projected/cca0e51a-2c91-4c8e-ac54-e682fe88271a-kube-api-access-vw56q\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:08.255721 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:08.255617    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"
Oct 31 19:44:08.255721 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:08.255637    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48" event={"ID":"cca0e51a-2c91-4c8e-ac54-e682fe88271a","Type":"ContainerDied","Data":"32be5d248ea4fb76b62aa8a30536d97042f586d3ac8d555914df96a51550b503"}
Oct 31 19:44:08.255721 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:08.255689    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32be5d248ea4fb76b62aa8a30536d97042f586d3ac8d555914df96a51550b503"
Oct 31 19:44:11.453039 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:11.452971    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76" podUID="c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99" containerName="payload-processing" containerID="cri-o://406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b" gracePeriod=25
Oct 31 19:44:11.544023 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:11.543977    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp" podUID="5f711e6a-d69b-4211-a01d-ba243b48501e" containerName="payload-pre-processing" containerID="cri-o://e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b" gracePeriod=25
Oct 31 19:44:16.014340 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:16.014305    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp" podUID="5f711e6a-d69b-4211-a01d-ba243b48501e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:44:16.014766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:16.014317    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76" podUID="c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:44:25.467966 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:44:25.467928    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:44:26.014163 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:26.014126    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp" podUID="5f711e6a-d69b-4211-a01d-ba243b48501e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:44:26.014339 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:26.014127    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76" podUID="c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:44:31.805916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.805892    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"
Oct 31 19:44:31.823393 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.823371    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"
Oct 31 19:44:31.877372 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.877351    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-kube-api-access-h5hf5\" (UniqueName: \"kubernetes.io/projected/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-kube-api-access-h5hf5\") pod \"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99\" (UID: \"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99\") "
Oct 31 19:44:31.877541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.877407    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-plugins-config-volume\") pod \"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99\" (UID: \"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99\") "
Oct 31 19:44:31.877541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.877432    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f711e6a-d69b-4211-a01d-ba243b48501e-kube-api-access-f6dvn\" (UniqueName: \"kubernetes.io/projected/5f711e6a-d69b-4211-a01d-ba243b48501e-kube-api-access-f6dvn\") pod \"5f711e6a-d69b-4211-a01d-ba243b48501e\" (UID: \"5f711e6a-d69b-4211-a01d-ba243b48501e\") "
Oct 31 19:44:31.877541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.877519    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f711e6a-d69b-4211-a01d-ba243b48501e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f711e6a-d69b-4211-a01d-ba243b48501e-plugins-config-volume\") pod \"5f711e6a-d69b-4211-a01d-ba243b48501e\" (UID: \"5f711e6a-d69b-4211-a01d-ba243b48501e\") "
Oct 31 19:44:31.877852 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.877829    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-plugins-config-volume" pod "c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99" (UID: "c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:31.877919 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.877903    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f711e6a-d69b-4211-a01d-ba243b48501e-plugins-config-volume" pod "5f711e6a-d69b-4211-a01d-ba243b48501e" (UID: "5f711e6a-d69b-4211-a01d-ba243b48501e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:31.879775 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.879752    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-kube-api-access-h5hf5" pod "c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99" (UID: "c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99"). InnerVolumeSpecName "kube-api-access-h5hf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:31.879881 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.879863    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f711e6a-d69b-4211-a01d-ba243b48501e-kube-api-access-f6dvn" pod "5f711e6a-d69b-4211-a01d-ba243b48501e" (UID: "5f711e6a-d69b-4211-a01d-ba243b48501e"). InnerVolumeSpecName "kube-api-access-f6dvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:31.979068 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.979035    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5hf5\" (UniqueName: \"kubernetes.io/projected/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-kube-api-access-h5hf5\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:31.979068 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.979061    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:31.979068 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.979071    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6dvn\" (UniqueName: \"kubernetes.io/projected/5f711e6a-d69b-4211-a01d-ba243b48501e-kube-api-access-f6dvn\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:31.979301 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:31.979079    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f711e6a-d69b-4211-a01d-ba243b48501e-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:32.377920 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.377833    4484 generic.go:356] "Generic (PLEG): container finished" podID="5f711e6a-d69b-4211-a01d-ba243b48501e" containerID="e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b" exitCode=0
Oct 31 19:44:32.377920 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.377888    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"
Oct 31 19:44:32.378135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.377922    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp" event={"ID":"5f711e6a-d69b-4211-a01d-ba243b48501e","Type":"ContainerDied","Data":"e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b"}
Oct 31 19:44:32.378135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.378001    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp" event={"ID":"5f711e6a-d69b-4211-a01d-ba243b48501e","Type":"ContainerDied","Data":"aa364212eda2d3a14cdc80dd086b8844709ca8c6ae178e0f1851577127ea96d0"}
Oct 31 19:44:32.378135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.378033    4484 scope.go:122] "RemoveContainer" containerID="e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b"
Oct 31 19:44:32.379519 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.379492    4484 generic.go:356] "Generic (PLEG): container finished" podID="c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99" containerID="406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b" exitCode=0
Oct 31 19:44:32.379660 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.379552    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"
Oct 31 19:44:32.379660 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.379574    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76" event={"ID":"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99","Type":"ContainerDied","Data":"406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b"}
Oct 31 19:44:32.379660 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.379612    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76" event={"ID":"c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99","Type":"ContainerDied","Data":"2b1464619e0bd30cd3834eadf0ed6a7d8ba8c5c7c36ce2d38eab6e88528932cc"}
Oct 31 19:44:32.386970 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.386951    4484 scope.go:122] "RemoveContainer" containerID="e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b"
Oct 31 19:44:32.387233 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:44:32.387212    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b\": container with ID starting with e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b not found: ID does not exist" containerID="e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b"
Oct 31 19:44:32.387282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.387242    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b"} err="failed to get container status \"e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b\": rpc error: code = NotFound desc = could not find container \"e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b\": container with ID starting with e35d2903904d51d323ecf981534377e95aee88a4410fb68929ee9cfa83779d4b not found: ID does not exist"
Oct 31 19:44:32.387282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.387260    4484 scope.go:122] "RemoveContainer" containerID="406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b"
Oct 31 19:44:32.394905 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.394888    4484 scope.go:122] "RemoveContainer" containerID="406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b"
Oct 31 19:44:32.395155 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:44:32.395137    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b\": container with ID starting with 406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b not found: ID does not exist" containerID="406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b"
Oct 31 19:44:32.395225 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.395163    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b"} err="failed to get container status \"406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b\": rpc error: code = NotFound desc = could not find container \"406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b\": container with ID starting with 406b82be89587635505378240271652b3c32b792daf80a4ae2a290dabf733b6b not found: ID does not exist"
Oct 31 19:44:32.546317 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.546285    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"]
Oct 31 19:44:32.551160 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.551132    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-88085e-5cb6ff8df5-8lj76"]
Oct 31 19:44:32.566405 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.566378    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:44:32.673299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.673229    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"]
Oct 31 19:44:32.678128 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:32.678103    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-88085e-778875b48c-h29mp"]
Oct 31 19:44:33.996281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:33.996242    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f711e6a-d69b-4211-a01d-ba243b48501e" path="/var/lib/kubelet/pods/5f711e6a-d69b-4211-a01d-ba243b48501e/volumes"
Oct 31 19:44:33.996873 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:33.996850    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99" path="/var/lib/kubelet/pods/c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99/volumes"
Oct 31 19:44:40.640148 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.640112    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"]
Oct 31 19:44:40.640603 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.640520    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99" containerName="payload-processing"
Oct 31 19:44:40.640603 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.640532    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99" containerName="payload-processing"
Oct 31 19:44:40.640603 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.640543    4484 cpu_manager.go:414] "containerMap: removing container" podUID="5f711e6a-d69b-4211-a01d-ba243b48501e" containerName="payload-pre-processing"
Oct 31 19:44:40.640603 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.640548    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f711e6a-d69b-4211-a01d-ba243b48501e" containerName="payload-pre-processing"
Oct 31 19:44:40.640603 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.640570    4484 cpu_manager.go:414] "containerMap: removing container" podUID="cca0e51a-2c91-4c8e-ac54-e682fe88271a" containerName="revoke-keys"
Oct 31 19:44:40.640603 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.640575    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca0e51a-2c91-4c8e-ac54-e682fe88271a" containerName="revoke-keys"
Oct 31 19:44:40.640795 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.640653    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cd62f3-4c84-4f3b-a0a8-39d9b1acde99" containerName="payload-processing"
Oct 31 19:44:40.640795 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.640662    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f711e6a-d69b-4211-a01d-ba243b48501e" containerName="payload-pre-processing"
Oct 31 19:44:40.640795 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.640672    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca0e51a-2c91-4c8e-ac54-e682fe88271a" containerName="revoke-keys"
Oct 31 19:44:40.643769 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.643753    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:40.701622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.701594    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"]
Oct 31 19:44:40.750058 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.750019    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:44:40.750058 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.750051    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:44:40.767575 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.767544    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt9k8\" (UniqueName: \"kubernetes.io/projected/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-kube-api-access-zt9k8\") pod \"maas-api-revoke-sub-14655279-p4n59\" (UID: \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:40.767772 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.767591    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14655279-p4n59\" (UID: \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:40.767772 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.767691    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14655279-p4n59\" (UID: \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:40.869226 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.869184    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14655279-p4n59\" (UID: \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:40.869415 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.869239    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14655279-p4n59\" (UID: \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:40.869415 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.869307    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt9k8\" (UniqueName: \"kubernetes.io/projected/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-kube-api-access-zt9k8\") pod \"maas-api-revoke-sub-14655279-p4n59\" (UID: \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:40.869993 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.869969    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14655279-p4n59\" (UID: \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:40.871987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.871964    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14655279-p4n59\" (UID: \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:40.930315 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.930247    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt9k8\" (UniqueName: \"kubernetes.io/projected/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-kube-api-access-zt9k8\") pod \"maas-api-revoke-sub-14655279-p4n59\" (UID: \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:40.956058 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:40.956017    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:41.337498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:41.337457    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"]
Oct 31 19:44:41.339543 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:44:41.339510    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c4bb76_1f53_4b2f_92a3_b6e0bba1a137.slice/crio-e71bdb487896d97dcccb780a11f2127a9b7fed1997f47802200c1fa31d200f98 WatchSource:0}: Error finding container e71bdb487896d97dcccb780a11f2127a9b7fed1997f47802200c1fa31d200f98: Status 404 returned error can't find the container with id e71bdb487896d97dcccb780a11f2127a9b7fed1997f47802200c1fa31d200f98
Oct 31 19:44:41.429553 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:41.429515    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59" event={"ID":"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137","Type":"ContainerStarted","Data":"e71bdb487896d97dcccb780a11f2127a9b7fed1997f47802200c1fa31d200f98"}
Oct 31 19:44:42.436159 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:42.436122    4484 generic.go:356] "Generic (PLEG): container finished" podID="40c4bb76-1f53-4b2f-92a3-b6e0bba1a137" containerID="7532f6275b11968bd88dd8302b1e086287b35b07bad04b46f6337af9f7720861" exitCode=0
Oct 31 19:44:42.436576 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:42.436207    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59" event={"ID":"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137","Type":"ContainerDied","Data":"7532f6275b11968bd88dd8302b1e086287b35b07bad04b46f6337af9f7720861"}
Oct 31 19:44:43.579369 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:43.579345    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:43.698055 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:43.697975    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-openshift-ca\") pod \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\" (UID: \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\") "
Oct 31 19:44:43.698055 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:43.698031    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-xks-ca\") pod \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\" (UID: \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\") "
Oct 31 19:44:43.698277 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:43.698199    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-kube-api-access-zt9k8\" (UniqueName: \"kubernetes.io/projected/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-kube-api-access-zt9k8\") pod \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\" (UID: \"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137\") "
Oct 31 19:44:43.698342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:43.698308    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-openshift-ca" pod "40c4bb76-1f53-4b2f-92a3-b6e0bba1a137" (UID: "40c4bb76-1f53-4b2f-92a3-b6e0bba1a137"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:43.698540 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:43.698522    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:43.700236 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:43.700212    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-xks-ca" pod "40c4bb76-1f53-4b2f-92a3-b6e0bba1a137" (UID: "40c4bb76-1f53-4b2f-92a3-b6e0bba1a137"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:44:43.700311 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:43.700260    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-kube-api-access-zt9k8" pod "40c4bb76-1f53-4b2f-92a3-b6e0bba1a137" (UID: "40c4bb76-1f53-4b2f-92a3-b6e0bba1a137"). InnerVolumeSpecName "kube-api-access-zt9k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:43.799967 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:43.799934    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt9k8\" (UniqueName: \"kubernetes.io/projected/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-kube-api-access-zt9k8\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:43.799967 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:43.799962    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:44.447398 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:44.447315    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"
Oct 31 19:44:44.447398 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:44.447332    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59" event={"ID":"40c4bb76-1f53-4b2f-92a3-b6e0bba1a137","Type":"ContainerDied","Data":"e71bdb487896d97dcccb780a11f2127a9b7fed1997f47802200c1fa31d200f98"}
Oct 31 19:44:44.447398 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:44.447373    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e71bdb487896d97dcccb780a11f2127a9b7fed1997f47802200c1fa31d200f98"
Oct 31 19:44:50.712992 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:50.712963    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"]
Oct 31 19:44:50.713454 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:50.713188    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerName="istio-proxy" containerID="cri-o://6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d" gracePeriod=30
Oct 31 19:44:51.367437 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.367402    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"]
Oct 31 19:44:51.367954 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.367935    4484 cpu_manager.go:414] "containerMap: removing container" podUID="40c4bb76-1f53-4b2f-92a3-b6e0bba1a137" containerName="revoke-keys"
Oct 31 19:44:51.368048 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.367956    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c4bb76-1f53-4b2f-92a3-b6e0bba1a137" containerName="revoke-keys"
Oct 31 19:44:51.368126 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.368057    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c4bb76-1f53-4b2f-92a3-b6e0bba1a137" containerName="revoke-keys"
Oct 31 19:44:51.371493 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.371458    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:51.431273 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.431239    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"]
Oct 31 19:44:51.472630 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.472600    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d270138-bg6c9\" (UID: \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:51.472820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.472651    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d270138-bg6c9\" (UID: \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:51.472820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.472786    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98lz7\" (UniqueName: \"kubernetes.io/projected/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-kube-api-access-98lz7\") pod \"maas-api-revoke-sub-9d270138-bg6c9\" (UID: \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:51.479192 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.479169    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:44:51.479314 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.479271    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:44:51.573748 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.573710    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d270138-bg6c9\" (UID: \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:51.573913 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.573774    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d270138-bg6c9\" (UID: \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:51.573913 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.573853    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98lz7\" (UniqueName: \"kubernetes.io/projected/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-kube-api-access-98lz7\") pod \"maas-api-revoke-sub-9d270138-bg6c9\" (UID: \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:51.574523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.574467    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d270138-bg6c9\" (UID: \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:51.576297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.576281    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d270138-bg6c9\" (UID: \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:51.633220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.633150    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98lz7\" (UniqueName: \"kubernetes.io/projected/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-kube-api-access-98lz7\") pod \"maas-api-revoke-sub-9d270138-bg6c9\" (UID: \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:51.684498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:51.684460    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:52.066160 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:52.066132    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"]
Oct 31 19:44:52.067247 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:44:52.067219    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d5f86a_5f66_4c7e_8e8e_e134ec6eefbd.slice/crio-50ccf1acbb566fd7cb93b0b5ad5db2157614e493f9c6af937f00b621a4445d32 WatchSource:0}: Error finding container 50ccf1acbb566fd7cb93b0b5ad5db2157614e493f9c6af937f00b621a4445d32: Status 404 returned error can't find the container with id 50ccf1acbb566fd7cb93b0b5ad5db2157614e493f9c6af937f00b621a4445d32
Oct 31 19:44:52.489435 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:52.489398    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9" event={"ID":"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd","Type":"ContainerStarted","Data":"27a311d1884fb600738b1ff94fee7fe256f360dfb0360eccb09e2c7b5490fccb"}
Oct 31 19:44:52.489632 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:52.489452    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9" event={"ID":"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd","Type":"ContainerStarted","Data":"50ccf1acbb566fd7cb93b0b5ad5db2157614e493f9c6af937f00b621a4445d32"}
Oct 31 19:44:52.617016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:52.616960    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9" podStartSLOduration=1.616945337 podStartE2EDuration="1.616945337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:52.614810442 +0000 UTC m=+5038.761260304" watchObservedRunningTime="2026-10-31 19:44:52.616945337 +0000 UTC m=+5038.763395200"
Oct 31 19:44:53.496458 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:53.496419    4484 generic.go:356] "Generic (PLEG): container finished" podID="59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd" containerID="27a311d1884fb600738b1ff94fee7fe256f360dfb0360eccb09e2c7b5490fccb" exitCode=0
Oct 31 19:44:53.496857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:53.496519    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9" event={"ID":"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd","Type":"ContainerDied","Data":"27a311d1884fb600738b1ff94fee7fe256f360dfb0360eccb09e2c7b5490fccb"}
Oct 31 19:44:54.634251 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:54.634226    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:54.805182 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:54.805091    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-xks-ca\") pod \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\" (UID: \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\") "
Oct 31 19:44:54.805374 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:54.805194    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-openshift-ca\") pod \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\" (UID: \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\") "
Oct 31 19:44:54.805374 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:54.805311    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-kube-api-access-98lz7\" (UniqueName: \"kubernetes.io/projected/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-kube-api-access-98lz7\") pod \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\" (UID: \"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd\") "
Oct 31 19:44:54.805569 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:54.805545    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-openshift-ca" pod "59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd" (UID: "59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:54.805692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:54.805677    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:54.807316 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:54.807294    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-xks-ca" pod "59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd" (UID: "59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:44:54.807443 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:54.807429    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-kube-api-access-98lz7" pod "59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd" (UID: "59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd"). InnerVolumeSpecName "kube-api-access-98lz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:54.906902 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:54.906867    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98lz7\" (UniqueName: \"kubernetes.io/projected/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-kube-api-access-98lz7\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:54.906902 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:54.906896    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:55.468256 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:44:55.468222    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:44:55.508179 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:55.508153    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"
Oct 31 19:44:55.508348 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:55.508180    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9" event={"ID":"59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd","Type":"ContainerDied","Data":"50ccf1acbb566fd7cb93b0b5ad5db2157614e493f9c6af937f00b621a4445d32"}
Oct 31 19:44:55.508348 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:55.508227    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50ccf1acbb566fd7cb93b0b5ad5db2157614e493f9c6af937f00b621a4445d32"
Oct 31 19:44:55.957774 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:55.957742    4484 scope.go:122] "RemoveContainer" containerID="384153cc820925dec13341644dcad1a6b1aef5c5042eedddc2801d5968a62096"
Oct 31 19:44:55.969767 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:44:55.969739    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384153cc820925dec13341644dcad1a6b1aef5c5042eedddc2801d5968a62096\": container with ID starting with 384153cc820925dec13341644dcad1a6b1aef5c5042eedddc2801d5968a62096 not found: ID does not exist" containerID="384153cc820925dec13341644dcad1a6b1aef5c5042eedddc2801d5968a62096"
Oct 31 19:44:55.973290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:55.973268    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 19:44:56.120110 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120077    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-podinfo\") pod \"be7b417c-a740-4b1d-ab49-467a568ef17f\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") "
Oct 31 19:44:56.120299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120141    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-certs\") pod \"be7b417c-a740-4b1d-ab49-467a568ef17f\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") "
Oct 31 19:44:56.120299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120164    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-envoy\") pod \"be7b417c-a740-4b1d-ab49-467a568ef17f\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") "
Oct 31 19:44:56.120299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120181    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-credential-socket\") pod \"be7b417c-a740-4b1d-ab49-467a568ef17f\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") "
Oct 31 19:44:56.120299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120200    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-kube-api-access-4m2f7\" (UniqueName: \"kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-kube-api-access-4m2f7\") pod \"be7b417c-a740-4b1d-ab49-467a568ef17f\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") "
Oct 31 19:44:56.120299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120246    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-token\" (UniqueName: \"kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-token\") pod \"be7b417c-a740-4b1d-ab49-467a568ef17f\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") "
Oct 31 19:44:56.120299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120286    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-data\") pod \"be7b417c-a740-4b1d-ab49-467a568ef17f\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") "
Oct 31 19:44:56.120678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120315    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be7b417c-a740-4b1d-ab49-467a568ef17f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7b417c-a740-4b1d-ab49-467a568ef17f-istiod-ca-cert\") pod \"be7b417c-a740-4b1d-ab49-467a568ef17f\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") "
Oct 31 19:44:56.120678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120421    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-socket\") pod \"be7b417c-a740-4b1d-ab49-467a568ef17f\" (UID: \"be7b417c-a740-4b1d-ab49-467a568ef17f\") "
Oct 31 19:44:56.120678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120550    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-certs" pod "be7b417c-a740-4b1d-ab49-467a568ef17f" (UID: "be7b417c-a740-4b1d-ab49-467a568ef17f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:56.120838 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120768    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-data" pod "be7b417c-a740-4b1d-ab49-467a568ef17f" (UID: "be7b417c-a740-4b1d-ab49-467a568ef17f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:56.120838 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120784    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.120952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120865    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-credential-socket" pod "be7b417c-a740-4b1d-ab49-467a568ef17f" (UID: "be7b417c-a740-4b1d-ab49-467a568ef17f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:56.121014 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.120944    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-socket" pod "be7b417c-a740-4b1d-ab49-467a568ef17f" (UID: "be7b417c-a740-4b1d-ab49-467a568ef17f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:56.121194 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.121174    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7b417c-a740-4b1d-ab49-467a568ef17f-istiod-ca-cert" pod "be7b417c-a740-4b1d-ab49-467a568ef17f" (UID: "be7b417c-a740-4b1d-ab49-467a568ef17f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:56.123080 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.123055    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-envoy" pod "be7b417c-a740-4b1d-ab49-467a568ef17f" (UID: "be7b417c-a740-4b1d-ab49-467a568ef17f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:56.123380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.123353    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-podinfo" pod "be7b417c-a740-4b1d-ab49-467a568ef17f" (UID: "be7b417c-a740-4b1d-ab49-467a568ef17f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:44:56.123545 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.123432    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-token" pod "be7b417c-a740-4b1d-ab49-467a568ef17f" (UID: "be7b417c-a740-4b1d-ab49-467a568ef17f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:56.123545 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.123455    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-kube-api-access-4m2f7" pod "be7b417c-a740-4b1d-ab49-467a568ef17f" (UID: "be7b417c-a740-4b1d-ab49-467a568ef17f"). InnerVolumeSpecName "kube-api-access-4m2f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:56.221506 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.221446    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.221506 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.221499    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.221506 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.221524    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.221771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.221533    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.221771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.221543    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m2f7\" (UniqueName: \"kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-kube-api-access-4m2f7\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.221771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.221551    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.221771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.221559    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7b417c-a740-4b1d-ab49-467a568ef17f-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.221771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.221568    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7b417c-a740-4b1d-ab49-467a568ef17f-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:44:56.515053 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.515019    4484 generic.go:356] "Generic (PLEG): container finished" podID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerID="6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d" exitCode=0
Oct 31 19:44:56.515252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.515064    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"
Oct 31 19:44:56.515252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.515104    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" event={"ID":"be7b417c-a740-4b1d-ab49-467a568ef17f","Type":"ContainerDied","Data":"6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d"}
Oct 31 19:44:56.515252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.515163    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2" event={"ID":"be7b417c-a740-4b1d-ab49-467a568ef17f","Type":"ContainerDied","Data":"23967608848fdd7dc5aafb8daa4678a3bd87e5a52cd7d6fa5f21f1ed715bb9d0"}
Oct 31 19:44:56.515252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.515195    4484 scope.go:122] "RemoveContainer" containerID="6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d"
Oct 31 19:44:56.524900 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.524882    4484 scope.go:122] "RemoveContainer" containerID="6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d"
Oct 31 19:44:56.525182 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:44:56.525162    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d\": container with ID starting with 6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d not found: ID does not exist" containerID="6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d"
Oct 31 19:44:56.525271 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.525188    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d"} err="failed to get container status \"6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d\": rpc error: code = NotFound desc = could not find container \"6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d\": container with ID starting with 6a1bf050645dd30041a52fa71ec7a826349950fc8c29ee5499d0a4c99001821d not found: ID does not exist"
Oct 31 19:44:56.686946 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.686909    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"]
Oct 31 19:44:56.737680 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:56.737652    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-88085e-openshift-default-848bbbb469-dbwm2"]
Oct 31 19:44:57.992644 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:44:57.992605    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" path="/var/lib/kubelet/pods/be7b417c-a740-4b1d-ab49-467a568ef17f/volumes"
Oct 31 19:45:00.162747 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.162716    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"]
Oct 31 19:45:00.163322 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.163304    4484 cpu_manager.go:414] "containerMap: removing container" podUID="59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd" containerName="revoke-keys"
Oct 31 19:45:00.163371 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.163324    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd" containerName="revoke-keys"
Oct 31 19:45:00.163419 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.163373    4484 cpu_manager.go:414] "containerMap: removing container" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerName="istio-proxy"
Oct 31 19:45:00.163419 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.163382    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerName="istio-proxy"
Oct 31 19:45:00.163419 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.163401    4484 cpu_manager.go:414] "containerMap: removing container" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerName="istio-proxy"
Oct 31 19:45:00.163419 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.163408    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerName="istio-proxy"
Oct 31 19:45:00.163583 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.163529    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerName="istio-proxy"
Oct 31 19:45:00.163583 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.163543    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7b417c-a740-4b1d-ab49-467a568ef17f" containerName="istio-proxy"
Oct 31 19:45:00.163583 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.163556    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd" containerName="revoke-keys"
Oct 31 19:45:00.168760 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.168740    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"]
Oct 31 19:45:00.168903 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.168863    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"
Oct 31 19:45:00.172499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.172461    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"
Oct 31 19:45:00.222518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.222461    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8mp6\""
Oct 31 19:45:00.264227 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.264195    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"]
Oct 31 19:45:00.275950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.275920    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"]
Oct 31 19:45:00.364523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.364492    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856da65f-5527-477c-84fa-b917d70cb60f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h\" (UID: \"856da65f-5527-477c-84fa-b917d70cb60f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"
Oct 31 19:45:00.364692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.364559    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6n6g\" (UniqueName: \"kubernetes.io/projected/a77aa675-9822-44eb-9373-7a473b9896a4-kube-api-access-h6n6g\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf\" (UID: \"a77aa675-9822-44eb-9373-7a473b9896a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"
Oct 31 19:45:00.364692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.364620    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hclg4\" (UniqueName: \"kubernetes.io/projected/856da65f-5527-477c-84fa-b917d70cb60f-kube-api-access-hclg4\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h\" (UID: \"856da65f-5527-477c-84fa-b917d70cb60f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"
Oct 31 19:45:00.364692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.364671    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77aa675-9822-44eb-9373-7a473b9896a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf\" (UID: \"a77aa675-9822-44eb-9373-7a473b9896a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"
Oct 31 19:45:00.465795 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.465715    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856da65f-5527-477c-84fa-b917d70cb60f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h\" (UID: \"856da65f-5527-477c-84fa-b917d70cb60f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"
Oct 31 19:45:00.465795 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.465787    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6n6g\" (UniqueName: \"kubernetes.io/projected/a77aa675-9822-44eb-9373-7a473b9896a4-kube-api-access-h6n6g\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf\" (UID: \"a77aa675-9822-44eb-9373-7a473b9896a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"
Oct 31 19:45:00.466004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.465807    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hclg4\" (UniqueName: \"kubernetes.io/projected/856da65f-5527-477c-84fa-b917d70cb60f-kube-api-access-hclg4\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h\" (UID: \"856da65f-5527-477c-84fa-b917d70cb60f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"
Oct 31 19:45:00.466004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.465825    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77aa675-9822-44eb-9373-7a473b9896a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf\" (UID: \"a77aa675-9822-44eb-9373-7a473b9896a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"
Oct 31 19:45:00.466399 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.466380    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856da65f-5527-477c-84fa-b917d70cb60f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h\" (UID: \"856da65f-5527-477c-84fa-b917d70cb60f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"
Oct 31 19:45:00.466447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.466414    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77aa675-9822-44eb-9373-7a473b9896a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf\" (UID: \"a77aa675-9822-44eb-9373-7a473b9896a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"
Oct 31 19:45:00.525007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.524973    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hclg4\" (UniqueName: \"kubernetes.io/projected/856da65f-5527-477c-84fa-b917d70cb60f-kube-api-access-hclg4\") pod \"maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h\" (UID: \"856da65f-5527-477c-84fa-b917d70cb60f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"
Oct 31 19:45:00.525161 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.525068    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6n6g\" (UniqueName: \"kubernetes.io/projected/a77aa675-9822-44eb-9373-7a473b9896a4-kube-api-access-h6n6g\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf\" (UID: \"a77aa675-9822-44eb-9373-7a473b9896a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"
Oct 31 19:45:00.783619 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.783588    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"
Oct 31 19:45:00.789643 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:00.789626    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"
Oct 31 19:45:01.163820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:01.163739    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"]
Oct 31 19:45:01.200160 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:45:01.200132    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856da65f_5527_477c_84fa_b917d70cb60f.slice/crio-11b603155d2289c21dd4212f816577e1b320dc4804aecbdfddcc79c9b6c3b3f7 WatchSource:0}: Error finding container 11b603155d2289c21dd4212f816577e1b320dc4804aecbdfddcc79c9b6c3b3f7: Status 404 returned error can't find the container with id 11b603155d2289c21dd4212f816577e1b320dc4804aecbdfddcc79c9b6c3b3f7
Oct 31 19:45:01.212778 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:01.212754    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"]
Oct 31 19:45:01.213298 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:45:01.213273    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77aa675_9822_44eb_9373_7a473b9896a4.slice/crio-5ae32bb0c80db2f51182c506d4ee71bd5bb9db109657aaa3766b81d919e47849 WatchSource:0}: Error finding container 5ae32bb0c80db2f51182c506d4ee71bd5bb9db109657aaa3766b81d919e47849: Status 404 returned error can't find the container with id 5ae32bb0c80db2f51182c506d4ee71bd5bb9db109657aaa3766b81d919e47849
Oct 31 19:45:01.542379 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:01.542346    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf" event={"ID":"a77aa675-9822-44eb-9373-7a473b9896a4","Type":"ContainerStarted","Data":"25b8cecc26cbbe06a774bc4fd195a3ae2347aa89c7f69ee665dba3afd5c39351"}
Oct 31 19:45:01.542610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:01.542407    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf" event={"ID":"a77aa675-9822-44eb-9373-7a473b9896a4","Type":"ContainerStarted","Data":"5ae32bb0c80db2f51182c506d4ee71bd5bb9db109657aaa3766b81d919e47849"}
Oct 31 19:45:01.543877 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:01.543855    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h" event={"ID":"856da65f-5527-477c-84fa-b917d70cb60f","Type":"ContainerStarted","Data":"2b8260b3ca8a1def7efdb29a4f622b011fc8cba07a9c23ac9b96d6738922e585"}
Oct 31 19:45:01.543968 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:01.543893    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h" event={"ID":"856da65f-5527-477c-84fa-b917d70cb60f","Type":"ContainerStarted","Data":"11b603155d2289c21dd4212f816577e1b320dc4804aecbdfddcc79c9b6c3b3f7"}
Oct 31 19:45:01.667361 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:01.667307    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf" podStartSLOduration=1.667290277 podStartE2EDuration="1.667290277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:01.665810708 +0000 UTC m=+5047.812260570" watchObservedRunningTime="2026-10-31 19:45:01.667290277 +0000 UTC m=+5047.813740120"
Oct 31 19:45:01.784159 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:01.784106    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h" podStartSLOduration=1.784090958 podStartE2EDuration="1.784090958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:01.782082543 +0000 UTC m=+5047.928532408" watchObservedRunningTime="2026-10-31 19:45:01.784090958 +0000 UTC m=+5047.930540819"
Oct 31 19:45:02.550746 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:02.550711    4484 generic.go:356] "Generic (PLEG): container finished" podID="856da65f-5527-477c-84fa-b917d70cb60f" containerID="2b8260b3ca8a1def7efdb29a4f622b011fc8cba07a9c23ac9b96d6738922e585" exitCode=60
Oct 31 19:45:02.551221 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:02.550804    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h" event={"ID":"856da65f-5527-477c-84fa-b917d70cb60f","Type":"ContainerDied","Data":"2b8260b3ca8a1def7efdb29a4f622b011fc8cba07a9c23ac9b96d6738922e585"}
Oct 31 19:45:02.551221 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:02.551113    4484 scope.go:122] "RemoveContainer" containerID="2b8260b3ca8a1def7efdb29a4f622b011fc8cba07a9c23ac9b96d6738922e585"
Oct 31 19:45:02.552552 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:02.552521    4484 generic.go:356] "Generic (PLEG): container finished" podID="a77aa675-9822-44eb-9373-7a473b9896a4" containerID="25b8cecc26cbbe06a774bc4fd195a3ae2347aa89c7f69ee665dba3afd5c39351" exitCode=60
Oct 31 19:45:02.552640 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:02.552614    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf" event={"ID":"a77aa675-9822-44eb-9373-7a473b9896a4","Type":"ContainerDied","Data":"25b8cecc26cbbe06a774bc4fd195a3ae2347aa89c7f69ee665dba3afd5c39351"}
Oct 31 19:45:02.552782 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:02.552764    4484 scope.go:122] "RemoveContainer" containerID="25b8cecc26cbbe06a774bc4fd195a3ae2347aa89c7f69ee665dba3afd5c39351"
Oct 31 19:45:03.559711 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:03.559679    4484 generic.go:356] "Generic (PLEG): container finished" podID="856da65f-5527-477c-84fa-b917d70cb60f" containerID="51948c88d883c80b37012e72b5ebb48d9751d86040485fc463fd15c66a6362ed" exitCode=60
Oct 31 19:45:03.560109 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:03.559774    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h" event={"ID":"856da65f-5527-477c-84fa-b917d70cb60f","Type":"ContainerDied","Data":"51948c88d883c80b37012e72b5ebb48d9751d86040485fc463fd15c66a6362ed"}
Oct 31 19:45:03.560109 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:03.559861    4484 scope.go:122] "RemoveContainer" containerID="2b8260b3ca8a1def7efdb29a4f622b011fc8cba07a9c23ac9b96d6738922e585"
Oct 31 19:45:03.560109 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:03.560103    4484 scope.go:122] "RemoveContainer" containerID="51948c88d883c80b37012e72b5ebb48d9751d86040485fc463fd15c66a6362ed"
Oct 31 19:45:03.560403 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:45:03.560375    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h_odh-ai-gateway-infra(856da65f-5527-477c-84fa-b917d70cb60f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h" podUID="856da65f-5527-477c-84fa-b917d70cb60f"
Oct 31 19:45:03.562183 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:03.562149    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf" event={"ID":"a77aa675-9822-44eb-9373-7a473b9896a4","Type":"ContainerStarted","Data":"098ac62fc48b0674c3338534ea5605d496ba8066aec8c50c7b8d9ff9afb4edc0"}
Oct 31 19:45:04.569622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:04.569592    4484 scope.go:122] "RemoveContainer" containerID="51948c88d883c80b37012e72b5ebb48d9751d86040485fc463fd15c66a6362ed"
Oct 31 19:45:04.570107 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:45:04.569887    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h_odh-ai-gateway-infra(856da65f-5527-477c-84fa-b917d70cb60f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h" podUID="856da65f-5527-477c-84fa-b917d70cb60f"
Oct 31 19:45:04.571162 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:04.571138    4484 generic.go:356] "Generic (PLEG): container finished" podID="a77aa675-9822-44eb-9373-7a473b9896a4" containerID="098ac62fc48b0674c3338534ea5605d496ba8066aec8c50c7b8d9ff9afb4edc0" exitCode=60
Oct 31 19:45:04.571302 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:04.571183    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf" event={"ID":"a77aa675-9822-44eb-9373-7a473b9896a4","Type":"ContainerDied","Data":"098ac62fc48b0674c3338534ea5605d496ba8066aec8c50c7b8d9ff9afb4edc0"}
Oct 31 19:45:04.571302 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:04.571236    4484 scope.go:122] "RemoveContainer" containerID="25b8cecc26cbbe06a774bc4fd195a3ae2347aa89c7f69ee665dba3afd5c39351"
Oct 31 19:45:04.571490 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:04.571451    4484 scope.go:122] "RemoveContainer" containerID="098ac62fc48b0674c3338534ea5605d496ba8066aec8c50c7b8d9ff9afb4edc0"
Oct 31 19:45:04.571726 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:45:04.571705    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf_odh-ai-gateway-infra(a77aa675-9822-44eb-9373-7a473b9896a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf" podUID="a77aa675-9822-44eb-9373-7a473b9896a4"
Oct 31 19:45:05.578141 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:05.578114    4484 scope.go:122] "RemoveContainer" containerID="098ac62fc48b0674c3338534ea5605d496ba8066aec8c50c7b8d9ff9afb4edc0"
Oct 31 19:45:05.578557 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:45:05.578331    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf_odh-ai-gateway-infra(a77aa675-9822-44eb-9373-7a473b9896a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf" podUID="a77aa675-9822-44eb-9373-7a473b9896a4"
Oct 31 19:45:06.411193 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:06.411157    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"]
Oct 31 19:45:07.189584 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:07.189552    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:45:15.986425 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:15.986388    4484 scope.go:122] "RemoveContainer" containerID="51948c88d883c80b37012e72b5ebb48d9751d86040485fc463fd15c66a6362ed"
Oct 31 19:45:16.634615 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:16.634529    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h" event={"ID":"856da65f-5527-477c-84fa-b917d70cb60f","Type":"ContainerStarted","Data":"7df31237f9c9f4f9049ccc293393c0113802f5b50f2f5c4c77dd5b41ca24c4e3"}
Oct 31 19:45:17.641043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:17.641007    4484 generic.go:356] "Generic (PLEG): container finished" podID="856da65f-5527-477c-84fa-b917d70cb60f" containerID="7df31237f9c9f4f9049ccc293393c0113802f5b50f2f5c4c77dd5b41ca24c4e3" exitCode=60
Oct 31 19:45:17.641533 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:17.641083    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h" event={"ID":"856da65f-5527-477c-84fa-b917d70cb60f","Type":"ContainerDied","Data":"7df31237f9c9f4f9049ccc293393c0113802f5b50f2f5c4c77dd5b41ca24c4e3"}
Oct 31 19:45:17.641533 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:17.641146    4484 scope.go:122] "RemoveContainer" containerID="51948c88d883c80b37012e72b5ebb48d9751d86040485fc463fd15c66a6362ed"
Oct 31 19:45:17.641633 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:17.641573    4484 scope.go:122] "RemoveContainer" containerID="7df31237f9c9f4f9049ccc293393c0113802f5b50f2f5c4c77dd5b41ca24c4e3"
Oct 31 19:45:17.641843 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:45:17.641822    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h_odh-ai-gateway-infra(856da65f-5527-477c-84fa-b917d70cb60f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h" podUID="856da65f-5527-477c-84fa-b917d70cb60f"
Oct 31 19:45:17.770958 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:17.770919    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"]
Oct 31 19:45:17.986601 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:17.986573    4484 scope.go:122] "RemoveContainer" containerID="098ac62fc48b0674c3338534ea5605d496ba8066aec8c50c7b8d9ff9afb4edc0"
Oct 31 19:45:18.650780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:18.650748    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf" event={"ID":"a77aa675-9822-44eb-9373-7a473b9896a4","Type":"ContainerStarted","Data":"d1e1dc15658f9b57bfbcc441a8bba14861bfe01dfc881fdd6647c4ddd12bc8d3"}
Oct 31 19:45:18.797657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:18.797626    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"
Oct 31 19:45:18.824351 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:18.824315    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856da65f-5527-477c-84fa-b917d70cb60f-kube-api-access-hclg4\" (UniqueName: \"kubernetes.io/projected/856da65f-5527-477c-84fa-b917d70cb60f-kube-api-access-hclg4\") pod \"856da65f-5527-477c-84fa-b917d70cb60f\" (UID: \"856da65f-5527-477c-84fa-b917d70cb60f\") "
Oct 31 19:45:18.824576 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:18.824359    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856da65f-5527-477c-84fa-b917d70cb60f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856da65f-5527-477c-84fa-b917d70cb60f-maas-api-service-ca\") pod \"856da65f-5527-477c-84fa-b917d70cb60f\" (UID: \"856da65f-5527-477c-84fa-b917d70cb60f\") "
Oct 31 19:45:18.824763 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:18.824742    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856da65f-5527-477c-84fa-b917d70cb60f-maas-api-service-ca" pod "856da65f-5527-477c-84fa-b917d70cb60f" (UID: "856da65f-5527-477c-84fa-b917d70cb60f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:18.826581 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:18.826559    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856da65f-5527-477c-84fa-b917d70cb60f-kube-api-access-hclg4" pod "856da65f-5527-477c-84fa-b917d70cb60f" (UID: "856da65f-5527-477c-84fa-b917d70cb60f"). InnerVolumeSpecName "kube-api-access-hclg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:18.925669 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:18.925586    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hclg4\" (UniqueName: \"kubernetes.io/projected/856da65f-5527-477c-84fa-b917d70cb60f-kube-api-access-hclg4\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:18.925669 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:18.925622    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856da65f-5527-477c-84fa-b917d70cb60f-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:19.657146 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:19.657122    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"
Oct 31 19:45:19.657634 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:19.657163    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h" event={"ID":"856da65f-5527-477c-84fa-b917d70cb60f","Type":"ContainerDied","Data":"11b603155d2289c21dd4212f816577e1b320dc4804aecbdfddcc79c9b6c3b3f7"}
Oct 31 19:45:19.657634 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:19.657226    4484 scope.go:122] "RemoveContainer" containerID="7df31237f9c9f4f9049ccc293393c0113802f5b50f2f5c4c77dd5b41ca24c4e3"
Oct 31 19:45:19.659051 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:19.659030    4484 generic.go:356] "Generic (PLEG): container finished" podID="a77aa675-9822-44eb-9373-7a473b9896a4" containerID="d1e1dc15658f9b57bfbcc441a8bba14861bfe01dfc881fdd6647c4ddd12bc8d3" exitCode=60
Oct 31 19:45:19.659197 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:19.659082    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf" event={"ID":"a77aa675-9822-44eb-9373-7a473b9896a4","Type":"ContainerDied","Data":"d1e1dc15658f9b57bfbcc441a8bba14861bfe01dfc881fdd6647c4ddd12bc8d3"}
Oct 31 19:45:19.659384 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:19.659369    4484 scope.go:122] "RemoveContainer" containerID="d1e1dc15658f9b57bfbcc441a8bba14861bfe01dfc881fdd6647c4ddd12bc8d3"
Oct 31 19:45:19.659660 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:45:19.659641    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf_odh-ai-gateway-infra(a77aa675-9822-44eb-9373-7a473b9896a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf" podUID="a77aa675-9822-44eb-9373-7a473b9896a4"
Oct 31 19:45:19.666252 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:19.666235    4484 scope.go:122] "RemoveContainer" containerID="098ac62fc48b0674c3338534ea5605d496ba8066aec8c50c7b8d9ff9afb4edc0"
Oct 31 19:45:19.842054 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:19.842023    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"]
Oct 31 19:45:19.853554 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:19.853522    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-da6ea7-29891265-gpl2h"]
Oct 31 19:45:19.905916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:19.905887    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"]
Oct 31 19:45:19.991949 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:19.991915    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856da65f-5527-477c-84fa-b917d70cb60f" path="/var/lib/kubelet/pods/856da65f-5527-477c-84fa-b917d70cb60f/volumes"
Oct 31 19:45:20.793654 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:20.793631    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"
Oct 31 19:45:20.845847 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:20.845816    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77aa675-9822-44eb-9373-7a473b9896a4-kube-api-access-h6n6g\" (UniqueName: \"kubernetes.io/projected/a77aa675-9822-44eb-9373-7a473b9896a4-kube-api-access-h6n6g\") pod \"a77aa675-9822-44eb-9373-7a473b9896a4\" (UID: \"a77aa675-9822-44eb-9373-7a473b9896a4\") "
Oct 31 19:45:20.845991 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:20.845879    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a77aa675-9822-44eb-9373-7a473b9896a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77aa675-9822-44eb-9373-7a473b9896a4-maas-api-service-ca\") pod \"a77aa675-9822-44eb-9373-7a473b9896a4\" (UID: \"a77aa675-9822-44eb-9373-7a473b9896a4\") "
Oct 31 19:45:20.846232 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:20.846213    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77aa675-9822-44eb-9373-7a473b9896a4-maas-api-service-ca" pod "a77aa675-9822-44eb-9373-7a473b9896a4" (UID: "a77aa675-9822-44eb-9373-7a473b9896a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:20.848212 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:20.848181    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77aa675-9822-44eb-9373-7a473b9896a4-kube-api-access-h6n6g" pod "a77aa675-9822-44eb-9373-7a473b9896a4" (UID: "a77aa675-9822-44eb-9373-7a473b9896a4"). InnerVolumeSpecName "kube-api-access-h6n6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:20.946807 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:20.946730    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6n6g\" (UniqueName: \"kubernetes.io/projected/a77aa675-9822-44eb-9373-7a473b9896a4-kube-api-access-h6n6g\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:20.946807 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:20.946756    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77aa675-9822-44eb-9373-7a473b9896a4-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:21.431107 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.431073    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54" podUID="9d0ffbc6-082f-48d3-8d04-e055df568533" containerName="main" containerID="cri-o://44ebdaae49f6bc9796154303ec6fbf573833ac37684099b92e0942ad3e9f3f11" gracePeriod=45
Oct 31 19:45:21.673536 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.673507    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf" event={"ID":"a77aa675-9822-44eb-9373-7a473b9896a4","Type":"ContainerDied","Data":"5ae32bb0c80db2f51182c506d4ee71bd5bb9db109657aaa3766b81d919e47849"}
Oct 31 19:45:21.673655 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.673563    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"
Oct 31 19:45:21.673655 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.673571    4484 scope.go:122] "RemoveContainer" containerID="d1e1dc15658f9b57bfbcc441a8bba14861bfe01dfc881fdd6647c4ddd12bc8d3"
Oct 31 19:45:21.674901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.674882    4484 generic.go:356] "Generic (PLEG): container finished" podID="9d0ffbc6-082f-48d3-8d04-e055df568533" containerID="44ebdaae49f6bc9796154303ec6fbf573833ac37684099b92e0942ad3e9f3f11" exitCode=0
Oct 31 19:45:21.674980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.674934    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54" event={"ID":"9d0ffbc6-082f-48d3-8d04-e055df568533","Type":"ContainerDied","Data":"44ebdaae49f6bc9796154303ec6fbf573833ac37684099b92e0942ad3e9f3f11"}
Oct 31 19:45:21.674980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.674954    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54" event={"ID":"9d0ffbc6-082f-48d3-8d04-e055df568533","Type":"ContainerDied","Data":"12f56ca12c0206361d8fda6c34ac1fe0e1c9dfc6c167e88c21cce546ac7bb1c7"}
Oct 31 19:45:21.674980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.674965    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12f56ca12c0206361d8fda6c34ac1fe0e1c9dfc6c167e88c21cce546ac7bb1c7"
Oct 31 19:45:21.681964 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.681942    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:45:21.754377 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.754343    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d0ffbc6-082f-48d3-8d04-e055df568533-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d0ffbc6-082f-48d3-8d04-e055df568533-tls-certs\") pod \"9d0ffbc6-082f-48d3-8d04-e055df568533\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") "
Oct 31 19:45:21.754602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.754390    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-model-cache\") pod \"9d0ffbc6-082f-48d3-8d04-e055df568533\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") "
Oct 31 19:45:21.754602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.754429    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0ffbc6-082f-48d3-8d04-e055df568533-kube-api-access-cgm84\" (UniqueName: \"kubernetes.io/projected/9d0ffbc6-082f-48d3-8d04-e055df568533-kube-api-access-cgm84\") pod \"9d0ffbc6-082f-48d3-8d04-e055df568533\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") "
Oct 31 19:45:21.754602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.754541    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-dshm\") pod \"9d0ffbc6-082f-48d3-8d04-e055df568533\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") "
Oct 31 19:45:21.754777 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.754613    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-home\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-home\") pod \"9d0ffbc6-082f-48d3-8d04-e055df568533\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") "
Oct 31 19:45:21.754777 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.754655    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-tmp-dir\") pod \"9d0ffbc6-082f-48d3-8d04-e055df568533\" (UID: \"9d0ffbc6-082f-48d3-8d04-e055df568533\") "
Oct 31 19:45:21.754777 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.754693    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-model-cache" pod "9d0ffbc6-082f-48d3-8d04-e055df568533" (UID: "9d0ffbc6-082f-48d3-8d04-e055df568533"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:21.754936 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.754875    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-home" pod "9d0ffbc6-082f-48d3-8d04-e055df568533" (UID: "9d0ffbc6-082f-48d3-8d04-e055df568533"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:21.754993 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.754960    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-tmp-dir" pod "9d0ffbc6-082f-48d3-8d04-e055df568533" (UID: "9d0ffbc6-082f-48d3-8d04-e055df568533"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:21.755099 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.755082    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:21.755159 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.755104    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:21.755159 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.755120    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:21.756731 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.756709    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-dshm" pod "9d0ffbc6-082f-48d3-8d04-e055df568533" (UID: "9d0ffbc6-082f-48d3-8d04-e055df568533"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:21.756813 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.756793    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0ffbc6-082f-48d3-8d04-e055df568533-tls-certs" pod "9d0ffbc6-082f-48d3-8d04-e055df568533" (UID: "9d0ffbc6-082f-48d3-8d04-e055df568533"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:45:21.756849 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.756813    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0ffbc6-082f-48d3-8d04-e055df568533-kube-api-access-cgm84" pod "9d0ffbc6-082f-48d3-8d04-e055df568533" (UID: "9d0ffbc6-082f-48d3-8d04-e055df568533"). InnerVolumeSpecName "kube-api-access-cgm84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:21.856239 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.856205    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d0ffbc6-082f-48d3-8d04-e055df568533-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:21.856239 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.856243    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgm84\" (UniqueName: \"kubernetes.io/projected/9d0ffbc6-082f-48d3-8d04-e055df568533-kube-api-access-cgm84\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:21.856658 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.856261    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d0ffbc6-082f-48d3-8d04-e055df568533-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:21.963656 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.963578    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"]
Oct 31 19:45:21.968810 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.968785    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f0e4e-29891265-lzptf"]
Oct 31 19:45:21.992630 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:21.992602    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a77aa675-9822-44eb-9373-7a473b9896a4" path="/var/lib/kubelet/pods/a77aa675-9822-44eb-9373-7a473b9896a4/volumes"
Oct 31 19:45:22.680645 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:22.680616    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"
Oct 31 19:45:22.850886 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:22.850853    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"]
Oct 31 19:45:22.901869 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:22.901838    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0f794f/rate-test-model-0f794f-kserve-65dc658d69-zgz54"]
Oct 31 19:45:23.366499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:23.366457    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:45:23.991998 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:23.991957    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0ffbc6-082f-48d3-8d04-e055df568533" path="/var/lib/kubelet/pods/9d0ffbc6-082f-48d3-8d04-e055df568533/volumes"
Oct 31 19:45:25.466890 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:45:25.466851    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:45:27.296449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:27.296415    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"]
Oct 31 19:45:34.996091 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:34.996058    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"]
Oct 31 19:45:37.684208 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:37.684130    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:45:39.996964 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:39.996932    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8" podUID="e4daaf71-c968-4d2a-8a18-9542333ac2c7" containerName="payload-pre-processing" containerID="cri-o://529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d" gracePeriod=25
Oct 31 19:45:42.316563 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.316515    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n" podUID="f9db77c4-3afe-4b27-b7f2-8268383dfadb" containerName="main" containerID="cri-o://000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf" gracePeriod=45
Oct 31 19:45:42.572037 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.571971    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:45:42.655029 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.654998    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-model-cache\") pod \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") "
Oct 31 19:45:42.655206 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.655093    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9db77c4-3afe-4b27-b7f2-8268383dfadb-kube-api-access-9777d\" (UniqueName: \"kubernetes.io/projected/f9db77c4-3afe-4b27-b7f2-8268383dfadb-kube-api-access-9777d\") pod \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") "
Oct 31 19:45:42.655206 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.655120    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-home\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-home\") pod \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") "
Oct 31 19:45:42.655206 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.655147    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tmp-dir\") pod \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") "
Oct 31 19:45:42.655206 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.655194    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tls-certs\") pod \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") "
Oct 31 19:45:42.655399 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.655233    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-dshm\") pod \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\" (UID: \"f9db77c4-3afe-4b27-b7f2-8268383dfadb\") "
Oct 31 19:45:42.655399 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.655283    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-model-cache" pod "f9db77c4-3afe-4b27-b7f2-8268383dfadb" (UID: "f9db77c4-3afe-4b27-b7f2-8268383dfadb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:42.655533 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.655386    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-home" pod "f9db77c4-3afe-4b27-b7f2-8268383dfadb" (UID: "f9db77c4-3afe-4b27-b7f2-8268383dfadb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:42.655533 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.655416    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tmp-dir" pod "f9db77c4-3afe-4b27-b7f2-8268383dfadb" (UID: "f9db77c4-3afe-4b27-b7f2-8268383dfadb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:42.655665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.655593    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:42.655665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.655607    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:42.655665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.655616    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:42.657589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.657560    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-dshm" pod "f9db77c4-3afe-4b27-b7f2-8268383dfadb" (UID: "f9db77c4-3afe-4b27-b7f2-8268383dfadb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:42.657709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.657595    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9db77c4-3afe-4b27-b7f2-8268383dfadb-kube-api-access-9777d" pod "f9db77c4-3afe-4b27-b7f2-8268383dfadb" (UID: "f9db77c4-3afe-4b27-b7f2-8268383dfadb"). InnerVolumeSpecName "kube-api-access-9777d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:42.657709 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.657597    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tls-certs" pod "f9db77c4-3afe-4b27-b7f2-8268383dfadb" (UID: "f9db77c4-3afe-4b27-b7f2-8268383dfadb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:45:42.756926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.756890    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9777d\" (UniqueName: \"kubernetes.io/projected/f9db77c4-3afe-4b27-b7f2-8268383dfadb-kube-api-access-9777d\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:42.756926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.756924    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9db77c4-3afe-4b27-b7f2-8268383dfadb-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:42.757140 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.756939    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9db77c4-3afe-4b27-b7f2-8268383dfadb-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:45:42.788164 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.788135    4484 generic.go:356] "Generic (PLEG): container finished" podID="f9db77c4-3afe-4b27-b7f2-8268383dfadb" containerID="000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf" exitCode=0
Oct 31 19:45:42.788361 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.788172    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n" event={"ID":"f9db77c4-3afe-4b27-b7f2-8268383dfadb","Type":"ContainerDied","Data":"000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf"}
Oct 31 19:45:42.788361 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.788189    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"
Oct 31 19:45:42.788361 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.788201    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n" event={"ID":"f9db77c4-3afe-4b27-b7f2-8268383dfadb","Type":"ContainerDied","Data":"c01d0478fd7ff0115776b0a4320a55594440208605f4761aed6b0c37825c1a71"}
Oct 31 19:45:42.788361 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.788218    4484 scope.go:122] "RemoveContainer" containerID="000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf"
Oct 31 19:45:42.797561 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.797542    4484 scope.go:122] "RemoveContainer" containerID="000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf"
Oct 31 19:45:42.797839 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:45:42.797820    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf\": container with ID starting with 000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf not found: ID does not exist" containerID="000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf"
Oct 31 19:45:42.797886 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.797847    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf"} err="failed to get container status \"000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf\": rpc error: code = NotFound desc = could not find container \"000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf\": container with ID starting with 000b1d3521b8ef45043e349b4065ea77f93e8f91444a03e1210933069a751fbf not found: ID does not exist"
Oct 31 19:45:42.956531 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.956433    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"]
Oct 31 19:45:42.961018 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:42.960992    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0f0e4e/rate-test-model-0f0e4e-kserve-7f9c7cb949-4k75n"]
Oct 31 19:45:43.445665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:43.445630    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8" podUID="e4daaf71-c968-4d2a-8a18-9542333ac2c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:45:43.993052 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:43.993020    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9db77c4-3afe-4b27-b7f2-8268383dfadb" path="/var/lib/kubelet/pods/f9db77c4-3afe-4b27-b7f2-8268383dfadb/volumes"
Oct 31 19:45:53.445335 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:53.445282    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8" podUID="e4daaf71-c968-4d2a-8a18-9542333ac2c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:45:54.939983 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:54.939953    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:45:54.947952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:45:54.947928    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:45:55.467179 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:45:55.467145    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:46:00.248554 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.248528    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"
Oct 31 19:46:00.324664 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.324588    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4daaf71-c968-4d2a-8a18-9542333ac2c7-kube-api-access-rnqm7\" (UniqueName: \"kubernetes.io/projected/e4daaf71-c968-4d2a-8a18-9542333ac2c7-kube-api-access-rnqm7\") pod \"e4daaf71-c968-4d2a-8a18-9542333ac2c7\" (UID: \"e4daaf71-c968-4d2a-8a18-9542333ac2c7\") "
Oct 31 19:46:00.324827 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.324708    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4daaf71-c968-4d2a-8a18-9542333ac2c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4daaf71-c968-4d2a-8a18-9542333ac2c7-plugins-config-volume\") pod \"e4daaf71-c968-4d2a-8a18-9542333ac2c7\" (UID: \"e4daaf71-c968-4d2a-8a18-9542333ac2c7\") "
Oct 31 19:46:00.325067 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.325044    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4daaf71-c968-4d2a-8a18-9542333ac2c7-plugins-config-volume" pod "e4daaf71-c968-4d2a-8a18-9542333ac2c7" (UID: "e4daaf71-c968-4d2a-8a18-9542333ac2c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:46:00.326779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.326758    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4daaf71-c968-4d2a-8a18-9542333ac2c7-kube-api-access-rnqm7" pod "e4daaf71-c968-4d2a-8a18-9542333ac2c7" (UID: "e4daaf71-c968-4d2a-8a18-9542333ac2c7"). InnerVolumeSpecName "kube-api-access-rnqm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:00.382160 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.382133    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:46:00.426331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.426305    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnqm7\" (UniqueName: \"kubernetes.io/projected/e4daaf71-c968-4d2a-8a18-9542333ac2c7-kube-api-access-rnqm7\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:00.426331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.426329    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4daaf71-c968-4d2a-8a18-9542333ac2c7-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:00.880253 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.880220    4484 generic.go:356] "Generic (PLEG): container finished" podID="e4daaf71-c968-4d2a-8a18-9542333ac2c7" containerID="529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d" exitCode=0
Oct 31 19:46:00.880505 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.880267    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"
Oct 31 19:46:00.880505 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.880276    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8" event={"ID":"e4daaf71-c968-4d2a-8a18-9542333ac2c7","Type":"ContainerDied","Data":"529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d"}
Oct 31 19:46:00.880505 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.880329    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8" event={"ID":"e4daaf71-c968-4d2a-8a18-9542333ac2c7","Type":"ContainerDied","Data":"6907accf002c2c5c5690a113850a8f264f4c4918cb208cb3141216447a3e3eb3"}
Oct 31 19:46:00.880505 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.880358    4484 scope.go:122] "RemoveContainer" containerID="529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d"
Oct 31 19:46:00.889743 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.889722    4484 scope.go:122] "RemoveContainer" containerID="529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d"
Oct 31 19:46:00.890013 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:46:00.889993    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d\": container with ID starting with 529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d not found: ID does not exist" containerID="529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d"
Oct 31 19:46:00.890059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:00.890022    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d"} err="failed to get container status \"529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d\": rpc error: code = NotFound desc = could not find container \"529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d\": container with ID starting with 529a9eb2ea32266c2ff19837f5712c88124651f29d71016ecacccc67ea4a765d not found: ID does not exist"
Oct 31 19:46:01.049745 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:01.049709    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"]
Oct 31 19:46:01.054162 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:01.054134    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0f794f-7bb97cb987-p8fd8"]
Oct 31 19:46:01.991971 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:01.991942    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4daaf71-c968-4d2a-8a18-9542333ac2c7" path="/var/lib/kubelet/pods/e4daaf71-c968-4d2a-8a18-9542333ac2c7/volumes"
Oct 31 19:46:16.723000 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:16.722963    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:46:18.925950 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:18.925920    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"]
Oct 31 19:46:25.467821 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:46:25.467784    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:46:30.056845 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:30.056814    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:46:33.945147 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:33.945113    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84" podUID="a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" containerName="main" containerID="cri-o://af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90" gracePeriod=45
Oct 31 19:46:34.196763 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.196709    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 19:46:34.244549 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.244516    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-home\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-home\") pod \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") "
Oct 31 19:46:34.244730 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.244566    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tls-certs\") pod \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") "
Oct 31 19:46:34.244730 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.244614    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tmp-dir\") pod \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") "
Oct 31 19:46:34.244730 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.244631    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-dshm\") pod \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") "
Oct 31 19:46:34.244730 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.244685    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-kube-api-access-gqxtl\" (UniqueName: \"kubernetes.io/projected/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-kube-api-access-gqxtl\") pod \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") "
Oct 31 19:46:34.244957 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.244852    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-home" pod "a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" (UID: "a2ff42d7-b4f9-4b25-b2b7-1e429f280aec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:34.244957 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.244903    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tmp-dir" pod "a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" (UID: "a2ff42d7-b4f9-4b25-b2b7-1e429f280aec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:34.245051 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.244993    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:34.245051 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.245012    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:34.247115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.247081    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-kube-api-access-gqxtl" pod "a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" (UID: "a2ff42d7-b4f9-4b25-b2b7-1e429f280aec"). InnerVolumeSpecName "kube-api-access-gqxtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:34.247115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.247093    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tls-certs" pod "a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" (UID: "a2ff42d7-b4f9-4b25-b2b7-1e429f280aec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:46:34.247247 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.247153    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-dshm" pod "a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" (UID: "a2ff42d7-b4f9-4b25-b2b7-1e429f280aec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:34.345780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.345741    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-model-cache\") pod \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\" (UID: \"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec\") "
Oct 31 19:46:34.345987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.345969    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:34.346068 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.345993    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqxtl\" (UniqueName: \"kubernetes.io/projected/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-kube-api-access-gqxtl\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:34.346068 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.346014    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:34.346153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.346063    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-model-cache" pod "a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" (UID: "a2ff42d7-b4f9-4b25-b2b7-1e429f280aec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:46:34.446911 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:34.446832    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:35.045976 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:35.045943    4484 generic.go:356] "Generic (PLEG): container finished" podID="a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" containerID="af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90" exitCode=0
Oct 31 19:46:35.046408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:35.046006    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"
Oct 31 19:46:35.046408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:35.046019    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84" event={"ID":"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec","Type":"ContainerDied","Data":"af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90"}
Oct 31 19:46:35.046408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:35.046061    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84" event={"ID":"a2ff42d7-b4f9-4b25-b2b7-1e429f280aec","Type":"ContainerDied","Data":"008804d04750bb650d6cd686d9245feb53806339e0ff7c877c414009c4f5a8fa"}
Oct 31 19:46:35.046408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:35.046080    4484 scope.go:122] "RemoveContainer" containerID="af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90"
Oct 31 19:46:35.055764 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:35.055746    4484 scope.go:122] "RemoveContainer" containerID="af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90"
Oct 31 19:46:35.056057 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:46:35.056039    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90\": container with ID starting with af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90 not found: ID does not exist" containerID="af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90"
Oct 31 19:46:35.056130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:35.056063    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90"} err="failed to get container status \"af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90\": rpc error: code = NotFound desc = could not find container \"af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90\": container with ID starting with af6ca825f1f754d23d02dd6f8ebbd395ed4491c44a0fa83dd43cc3a5f18d5b90 not found: ID does not exist"
Oct 31 19:46:35.175980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:35.175916    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84" podUID="a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 31 19:46:35.216016 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:35.215979    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"]
Oct 31 19:46:35.266856 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:35.266819    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-da6ea7/distinct-2-w1-da6ea7-kserve-646bcdbb97-xkx84"]
Oct 31 19:46:35.992643 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:35.992607    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" path="/var/lib/kubelet/pods/a2ff42d7-b4f9-4b25-b2b7-1e429f280aec/volumes"
Oct 31 19:46:37.701883 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:37.701848    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"]
Oct 31 19:46:37.817845 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:37.817814    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"]
Oct 31 19:46:42.702401 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:42.702355    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc" podUID="486fd098-2d6b-4a72-a28a-b7203b42bb61" containerName="payload-processing" containerID="cri-o://0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f" gracePeriod=25
Oct 31 19:46:42.818823 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:42.818781    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6" podUID="0ff209d0-91a6-4b4e-9fd5-fe7258c12c96" containerName="payload-pre-processing" containerID="cri-o://5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532" gracePeriod=25
Oct 31 19:46:45.965935 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.965900    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"]
Oct 31 19:46:45.966356 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966340    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f9db77c4-3afe-4b27-b7f2-8268383dfadb" containerName="main"
Oct 31 19:46:45.966403 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966359    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9db77c4-3afe-4b27-b7f2-8268383dfadb" containerName="main"
Oct 31 19:46:45.966403 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966371    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" containerName="main"
Oct 31 19:46:45.966403 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966376    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" containerName="main"
Oct 31 19:46:45.966403 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966391    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a77aa675-9822-44eb-9373-7a473b9896a4" containerName="cleanup"
Oct 31 19:46:45.966403 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966398    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77aa675-9822-44eb-9373-7a473b9896a4" containerName="cleanup"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966409    4484 cpu_manager.go:414] "containerMap: removing container" podUID="e4daaf71-c968-4d2a-8a18-9542333ac2c7" containerName="payload-pre-processing"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966414    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4daaf71-c968-4d2a-8a18-9542333ac2c7" containerName="payload-pre-processing"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966427    4484 cpu_manager.go:414] "containerMap: removing container" podUID="856da65f-5527-477c-84fa-b917d70cb60f" containerName="cleanup"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966436    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856da65f-5527-477c-84fa-b917d70cb60f" containerName="cleanup"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966452    4484 cpu_manager.go:414] "containerMap: removing container" podUID="9d0ffbc6-082f-48d3-8d04-e055df568533" containerName="main"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966457    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0ffbc6-082f-48d3-8d04-e055df568533" containerName="main"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966462    4484 cpu_manager.go:414] "containerMap: removing container" podUID="856da65f-5527-477c-84fa-b917d70cb60f" containerName="cleanup"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966466    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856da65f-5527-477c-84fa-b917d70cb60f" containerName="cleanup"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966485    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a77aa675-9822-44eb-9373-7a473b9896a4" containerName="cleanup"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966489    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77aa675-9822-44eb-9373-7a473b9896a4" containerName="cleanup"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966501    4484 cpu_manager.go:414] "containerMap: removing container" podUID="856da65f-5527-477c-84fa-b917d70cb60f" containerName="cleanup"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966507    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856da65f-5527-477c-84fa-b917d70cb60f" containerName="cleanup"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966566    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4daaf71-c968-4d2a-8a18-9542333ac2c7" containerName="payload-pre-processing"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966573    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="856da65f-5527-477c-84fa-b917d70cb60f" containerName="cleanup"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966581    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77aa675-9822-44eb-9373-7a473b9896a4" containerName="cleanup"
Oct 31 19:46:45.966588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966590    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0ffbc6-082f-48d3-8d04-e055df568533" containerName="main"
Oct 31 19:46:45.967068 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966600    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="856da65f-5527-477c-84fa-b917d70cb60f" containerName="cleanup"
Oct 31 19:46:45.967068 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966606    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77aa675-9822-44eb-9373-7a473b9896a4" containerName="cleanup"
Oct 31 19:46:45.967068 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966612    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="856da65f-5527-477c-84fa-b917d70cb60f" containerName="cleanup"
Oct 31 19:46:45.967068 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966618    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9db77c4-3afe-4b27-b7f2-8268383dfadb" containerName="main"
Oct 31 19:46:45.967068 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.966623    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ff42d7-b4f9-4b25-b2b7-1e429f280aec" containerName="main"
Oct 31 19:46:45.969850 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:45.969834    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.023865 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.023837    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-da6ea7-openshift-default-dockercfg-z7k66\""
Oct 31 19:46:46.045835 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.045796    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4ph\" (UniqueName: \"kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-kube-api-access-vx4ph\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.045948 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.045846    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.045948 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.045914    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.046026 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.045980    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.046026 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.046013    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.046099 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.046084    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.046138 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.046131    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.046191 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.046146    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d872427b-ff64-435c-9486-ec315dfc11cb-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.046284 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.046265    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d872427b-ff64-435c-9486-ec315dfc11cb-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.076707 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.076676    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"]
Oct 31 19:46:46.146767 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.146737    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d872427b-ff64-435c-9486-ec315dfc11cb-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.146938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.146776    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4ph\" (UniqueName: \"kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-kube-api-access-vx4ph\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.146938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.146818    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.146938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.146846    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.146938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.146869    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.146938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.146895    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.146938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.146918    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.147345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.147084    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.147345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.147127    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d872427b-ff64-435c-9486-ec315dfc11cb-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.147345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.147254    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-credential-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.147345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.147290    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-socket\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.147590 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.147439    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-certs\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.147590 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.147515    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-data\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.148180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.148158    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d872427b-ff64-435c-9486-ec315dfc11cb-istiod-ca-cert\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.149196 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.149175    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-envoy\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.149465 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.149447    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d872427b-ff64-435c-9486-ec315dfc11cb-istio-podinfo\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.207074 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.207040    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-istio-token\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.207226 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.207089    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4ph\" (UniqueName: \"kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-kube-api-access-vx4ph\") pod \"e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") " pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.282329 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.282302    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:46.715921 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.715895    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"]
Oct 31 19:46:46.716572 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:46:46.716544    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd872427b_ff64_435c_9486_ec315dfc11cb.slice/crio-05f0cadee858cddb8a6ba0a5d838c168bc17b8f5d05aa185214482625cdc4a39 WatchSource:0}: Error finding container 05f0cadee858cddb8a6ba0a5d838c168bc17b8f5d05aa185214482625cdc4a39: Status 404 returned error can't find the container with id 05f0cadee858cddb8a6ba0a5d838c168bc17b8f5d05aa185214482625cdc4a39
Oct 31 19:46:46.718735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.718703    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:46:46.718790 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.718767    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:46:46.718835 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:46.718794    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:46:47.104093 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:47.104054    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" event={"ID":"d872427b-ff64-435c-9486-ec315dfc11cb","Type":"ContainerStarted","Data":"7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a"}
Oct 31 19:46:47.104593 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:47.104108    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" event={"ID":"d872427b-ff64-435c-9486-ec315dfc11cb","Type":"ContainerStarted","Data":"05f0cadee858cddb8a6ba0a5d838c168bc17b8f5d05aa185214482625cdc4a39"}
Oct 31 19:46:47.143030 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:47.142984    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc" podUID="486fd098-2d6b-4a72-a28a-b7203b42bb61" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:46:47.143030 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:47.143021    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6" podUID="0ff209d0-91a6-4b4e-9fd5-fe7258c12c96" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:46:47.285498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:47.285084    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:47.285498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:47.285212    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:46:47.285498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:47.285254    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 19:46:47.321981 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:47.321929    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" podStartSLOduration=2.321913847 podStartE2EDuration="2.321913847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:47.285034607 +0000 UTC m=+5153.431484473" watchObservedRunningTime="2026-10-31 19:46:47.321913847 +0000 UTC m=+5153.468363711"
Oct 31 19:46:48.285925 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:48.285892    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:46:48.286467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:48.286101    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 19:46:49.285979 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:49.285940    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:46:49.286643 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:49.286608    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 19:46:50.284871 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.284837    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:46:50.285152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.285126    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 19:46:50.346007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.345969    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"]
Oct 31 19:46:50.346626 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.346598    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d" podUID="0575483e-549c-4b63-bc9f-72218b8d72d8" containerName="maas-api" containerID="cri-o://2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f" gracePeriod=30
Oct 31 19:46:50.647541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.647519    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 19:46:50.708949 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.708919    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-metrics-certs\") pod \"0575483e-549c-4b63-bc9f-72218b8d72d8\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") "
Oct 31 19:46:50.709124 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.708987    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0575483e-549c-4b63-bc9f-72218b8d72d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0575483e-549c-4b63-bc9f-72218b8d72d8-openshift-service-ca\") pod \"0575483e-549c-4b63-bc9f-72218b8d72d8\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") "
Oct 31 19:46:50.709124 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.709014    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-maas-api-tls\") pod \"0575483e-549c-4b63-bc9f-72218b8d72d8\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") "
Oct 31 19:46:50.709124 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.709047    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0575483e-549c-4b63-bc9f-72218b8d72d8-kube-api-access-5lbpv\" (UniqueName: \"kubernetes.io/projected/0575483e-549c-4b63-bc9f-72218b8d72d8-kube-api-access-5lbpv\") pod \"0575483e-549c-4b63-bc9f-72218b8d72d8\" (UID: \"0575483e-549c-4b63-bc9f-72218b8d72d8\") "
Oct 31 19:46:50.709345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.709322    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0575483e-549c-4b63-bc9f-72218b8d72d8-openshift-service-ca" pod "0575483e-549c-4b63-bc9f-72218b8d72d8" (UID: "0575483e-549c-4b63-bc9f-72218b8d72d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:46:50.711365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.711339    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-maas-api-tls" pod "0575483e-549c-4b63-bc9f-72218b8d72d8" (UID: "0575483e-549c-4b63-bc9f-72218b8d72d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:46:50.711509 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.711401    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0575483e-549c-4b63-bc9f-72218b8d72d8-kube-api-access-5lbpv" pod "0575483e-549c-4b63-bc9f-72218b8d72d8" (UID: "0575483e-549c-4b63-bc9f-72218b8d72d8"). InnerVolumeSpecName "kube-api-access-5lbpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:46:50.711568 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.711526    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-metrics-certs" pod "0575483e-549c-4b63-bc9f-72218b8d72d8" (UID: "0575483e-549c-4b63-bc9f-72218b8d72d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:46:50.810048 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.809972    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lbpv\" (UniqueName: \"kubernetes.io/projected/0575483e-549c-4b63-bc9f-72218b8d72d8-kube-api-access-5lbpv\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:50.810048 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.809999    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:50.810048 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.810009    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0575483e-549c-4b63-bc9f-72218b8d72d8-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:50.810048 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:50.810019    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0575483e-549c-4b63-bc9f-72218b8d72d8-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:46:51.124739 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:51.124610    4484 generic.go:356] "Generic (PLEG): container finished" podID="0575483e-549c-4b63-bc9f-72218b8d72d8" containerID="2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f" exitCode=0
Oct 31 19:46:51.124739 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:51.124665    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"
Oct 31 19:46:51.125282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:51.124665    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d" event={"ID":"0575483e-549c-4b63-bc9f-72218b8d72d8","Type":"ContainerDied","Data":"2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f"}
Oct 31 19:46:51.125282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:51.125043    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d" event={"ID":"0575483e-549c-4b63-bc9f-72218b8d72d8","Type":"ContainerDied","Data":"a58bc63adc6b6040725350c5874b8e88fd0834a74414868b40ccabb16e362c6a"}
Oct 31 19:46:51.125282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:51.125079    4484 scope.go:122] "RemoveContainer" containerID="2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f"
Oct 31 19:46:51.135420 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:51.135400    4484 scope.go:122] "RemoveContainer" containerID="2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f"
Oct 31 19:46:51.135723 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:46:51.135678    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f\": container with ID starting with 2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f not found: ID does not exist" containerID="2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f"
Oct 31 19:46:51.135820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:51.135711    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f"} err="failed to get container status \"2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f\": rpc error: code = NotFound desc = could not find container \"2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f\": container with ID starting with 2d6d85dded032d5f577ca3c4afba9c9cf592232fffbaed0949e6f9a618c7e54f not found: ID does not exist"
Oct 31 19:46:51.284926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:51.284889    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:46:51.285207 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:51.285178    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 19:46:51.295677 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:51.295647    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"]
Oct 31 19:46:51.346652 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:51.346623    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-da6ea7-75b6b98d7f-2fq5d"]
Oct 31 19:46:51.993337 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:51.993293    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0575483e-549c-4b63-bc9f-72218b8d72d8" path="/var/lib/kubelet/pods/0575483e-549c-4b63-bc9f-72218b8d72d8/volumes"
Oct 31 19:46:52.284987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:52.284896    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 19:46:52.285506 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:52.285220    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 19:46:52.977820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:52.977791    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:46:54.324620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:54.324577    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:46:54.325101 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:54.324650    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:55.366696 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:55.366650    4484 patch_prober.go:29] interesting pod/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:46:55.367250 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:55.366743    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:46:55.491112 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:46:55.491072    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:46:55.504279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:55.504245    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:55.505096 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:55.504960    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:55.523064 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:55.523032    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:46:57.142707 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:57.142671    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6" podUID="0ff209d0-91a6-4b4e-9fd5-fe7258c12c96" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:46:57.143092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:57.142671    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc" podUID="486fd098-2d6b-4a72-a28a-b7203b42bb61" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:46:59.146651 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:59.146615    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"]
Oct 31 19:46:59.147078 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:46:59.146823    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" containerName="istio-proxy" containerID="cri-o://7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a" gracePeriod=30
Oct 31 19:47:02.186632 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:02.186599    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-z57jb_5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a/limitador/0.log"
Oct 31 19:47:02.959372 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:02.959064    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"
Oct 31 19:47:03.022940 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.022912    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486fd098-2d6b-4a72-a28a-b7203b42bb61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486fd098-2d6b-4a72-a28a-b7203b42bb61-plugins-config-volume\") pod \"486fd098-2d6b-4a72-a28a-b7203b42bb61\" (UID: \"486fd098-2d6b-4a72-a28a-b7203b42bb61\") "
Oct 31 19:47:03.023126 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.022985    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486fd098-2d6b-4a72-a28a-b7203b42bb61-kube-api-access-6dzqg\" (UniqueName: \"kubernetes.io/projected/486fd098-2d6b-4a72-a28a-b7203b42bb61-kube-api-access-6dzqg\") pod \"486fd098-2d6b-4a72-a28a-b7203b42bb61\" (UID: \"486fd098-2d6b-4a72-a28a-b7203b42bb61\") "
Oct 31 19:47:03.023317 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.023291    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486fd098-2d6b-4a72-a28a-b7203b42bb61-plugins-config-volume" pod "486fd098-2d6b-4a72-a28a-b7203b42bb61" (UID: "486fd098-2d6b-4a72-a28a-b7203b42bb61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:03.025547 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.025505    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486fd098-2d6b-4a72-a28a-b7203b42bb61-kube-api-access-6dzqg" pod "486fd098-2d6b-4a72-a28a-b7203b42bb61" (UID: "486fd098-2d6b-4a72-a28a-b7203b42bb61"). InnerVolumeSpecName "kube-api-access-6dzqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:03.050222 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.050197    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"
Oct 31 19:47:03.123893 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.123857    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-plugins-config-volume\") pod \"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96\" (UID: \"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96\") "
Oct 31 19:47:03.124097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.123922    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-kube-api-access-6ns6r\" (UniqueName: \"kubernetes.io/projected/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-kube-api-access-6ns6r\") pod \"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96\" (UID: \"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96\") "
Oct 31 19:47:03.124218 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.124199    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dzqg\" (UniqueName: \"kubernetes.io/projected/486fd098-2d6b-4a72-a28a-b7203b42bb61-kube-api-access-6dzqg\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:03.124278 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.124224    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486fd098-2d6b-4a72-a28a-b7203b42bb61-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:03.124322 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.124282    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-plugins-config-volume" pod "0ff209d0-91a6-4b4e-9fd5-fe7258c12c96" (UID: "0ff209d0-91a6-4b4e-9fd5-fe7258c12c96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:03.126263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.126240    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-kube-api-access-6ns6r" pod "0ff209d0-91a6-4b4e-9fd5-fe7258c12c96" (UID: "0ff209d0-91a6-4b4e-9fd5-fe7258c12c96"). InnerVolumeSpecName "kube-api-access-6ns6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:03.185006 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.184971    4484 generic.go:356] "Generic (PLEG): container finished" podID="0ff209d0-91a6-4b4e-9fd5-fe7258c12c96" containerID="5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532" exitCode=0
Oct 31 19:47:03.185267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.185038    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"
Oct 31 19:47:03.185267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.185047    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6" event={"ID":"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96","Type":"ContainerDied","Data":"5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532"}
Oct 31 19:47:03.185267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.185108    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6" event={"ID":"0ff209d0-91a6-4b4e-9fd5-fe7258c12c96","Type":"ContainerDied","Data":"4f504e987aa808bb9564404e960eeaa3c71faca71aaea3ec10405e6672d85c08"}
Oct 31 19:47:03.185267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.185142    4484 scope.go:122] "RemoveContainer" containerID="5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532"
Oct 31 19:47:03.186578 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.186557    4484 generic.go:356] "Generic (PLEG): container finished" podID="486fd098-2d6b-4a72-a28a-b7203b42bb61" containerID="0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f" exitCode=0
Oct 31 19:47:03.186699 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.186605    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"
Oct 31 19:47:03.186699 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.186637    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc" event={"ID":"486fd098-2d6b-4a72-a28a-b7203b42bb61","Type":"ContainerDied","Data":"0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f"}
Oct 31 19:47:03.186699 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.186681    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc" event={"ID":"486fd098-2d6b-4a72-a28a-b7203b42bb61","Type":"ContainerDied","Data":"30cb21c75a95c842a361992f1c5666549efab74cb1e7d86c4984d5b59cf990e2"}
Oct 31 19:47:03.197219 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.197187    4484 scope.go:122] "RemoveContainer" containerID="5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532"
Oct 31 19:47:03.197616 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:47:03.197595    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532\": container with ID starting with 5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532 not found: ID does not exist" containerID="5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532"
Oct 31 19:47:03.197737 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.197620    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532"} err="failed to get container status \"5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532\": rpc error: code = NotFound desc = could not find container \"5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532\": container with ID starting with 5270df5921a0c7e2f76919d8e564310a164d4c7b48ad7e8aab6d2700c871b532 not found: ID does not exist"
Oct 31 19:47:03.197737 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.197642    4484 scope.go:122] "RemoveContainer" containerID="0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f"
Oct 31 19:47:03.207594 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.207570    4484 scope.go:122] "RemoveContainer" containerID="0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f"
Oct 31 19:47:03.207939 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:47:03.207921    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f\": container with ID starting with 0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f not found: ID does not exist" containerID="0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f"
Oct 31 19:47:03.207995 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.207948    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f"} err="failed to get container status \"0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f\": rpc error: code = NotFound desc = could not find container \"0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f\": container with ID starting with 0062aa0c45ac7ecb1881b278bfb0187ae495c1eaca8989f075d4bf7b50112d5f not found: ID does not exist"
Oct 31 19:47:03.225009 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.224957    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ns6r\" (UniqueName: \"kubernetes.io/projected/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-kube-api-access-6ns6r\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:03.225009 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.224980    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:03.416736 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.416704    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"]
Oct 31 19:47:03.418500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.418454    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-da6ea7-5f5ffcfc85-vvrzc"]
Oct 31 19:47:03.528938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.528905    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"]
Oct 31 19:47:03.534215 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.534176    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-da6ea7-5df68c66b7-j9xf6"]
Oct 31 19:47:03.739659 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.739635    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c8b7d8cc9-rzk2k_ea33ee05-e391-4d64-a57a-039b32d83cd2/manager/1.log"
Oct 31 19:47:03.992148 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.992119    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff209d0-91a6-4b4e-9fd5-fe7258c12c96" path="/var/lib/kubelet/pods/0ff209d0-91a6-4b4e-9fd5-fe7258c12c96/volumes"
Oct 31 19:47:03.992518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:03.992504    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486fd098-2d6b-4a72-a28a-b7203b42bb61" path="/var/lib/kubelet/pods/486fd098-2d6b-4a72-a28a-b7203b42bb61/volumes"
Oct 31 19:47:04.096155 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.096124    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hf75l_30e22b48-ac4a-4c9c-8154-79b160d5c529/manager/0.log"
Oct 31 19:47:04.395510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.395488    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:47:04.536375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.536303    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-istio-token\" (UniqueName: \"kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-istio-token\") pod \"d872427b-ff64-435c-9486-ec315dfc11cb\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") "
Oct 31 19:47:04.536375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.536350    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-envoy\") pod \"d872427b-ff64-435c-9486-ec315dfc11cb\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") "
Oct 31 19:47:04.536618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.536401    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-certs\") pod \"d872427b-ff64-435c-9486-ec315dfc11cb\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") "
Oct 31 19:47:04.536618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.536429    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d872427b-ff64-435c-9486-ec315dfc11cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d872427b-ff64-435c-9486-ec315dfc11cb-istiod-ca-cert\") pod \"d872427b-ff64-435c-9486-ec315dfc11cb\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") "
Oct 31 19:47:04.536618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.536528    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-kube-api-access-vx4ph\" (UniqueName: \"kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-kube-api-access-vx4ph\") pod \"d872427b-ff64-435c-9486-ec315dfc11cb\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") "
Oct 31 19:47:04.536618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.536562    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-data\") pod \"d872427b-ff64-435c-9486-ec315dfc11cb\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") "
Oct 31 19:47:04.536618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.536588    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-socket\") pod \"d872427b-ff64-435c-9486-ec315dfc11cb\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") "
Oct 31 19:47:04.536618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.536614    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-credential-socket\") pod \"d872427b-ff64-435c-9486-ec315dfc11cb\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") "
Oct 31 19:47:04.536940 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.536676    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d872427b-ff64-435c-9486-ec315dfc11cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d872427b-ff64-435c-9486-ec315dfc11cb-istio-podinfo\") pod \"d872427b-ff64-435c-9486-ec315dfc11cb\" (UID: \"d872427b-ff64-435c-9486-ec315dfc11cb\") "
Oct 31 19:47:04.536940 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.536764    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-certs" pod "d872427b-ff64-435c-9486-ec315dfc11cb" (UID: "d872427b-ff64-435c-9486-ec315dfc11cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:04.536940 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.536871    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-data" pod "d872427b-ff64-435c-9486-ec315dfc11cb" (UID: "d872427b-ff64-435c-9486-ec315dfc11cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:04.536940 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.536893    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d872427b-ff64-435c-9486-ec315dfc11cb-istiod-ca-cert" pod "d872427b-ff64-435c-9486-ec315dfc11cb" (UID: "d872427b-ff64-435c-9486-ec315dfc11cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:04.537167 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.537061    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-socket" pod "d872427b-ff64-435c-9486-ec315dfc11cb" (UID: "d872427b-ff64-435c-9486-ec315dfc11cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:04.537167 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.537063    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.537167 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.537085    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-credential-socket" pod "d872427b-ff64-435c-9486-ec315dfc11cb" (UID: "d872427b-ff64-435c-9486-ec315dfc11cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:04.537167 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.537098    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.537167 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.537110    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d872427b-ff64-435c-9486-ec315dfc11cb-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.538724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.538702    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-istio-token" pod "d872427b-ff64-435c-9486-ec315dfc11cb" (UID: "d872427b-ff64-435c-9486-ec315dfc11cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:04.538990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.538967    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d872427b-ff64-435c-9486-ec315dfc11cb-istio-podinfo" pod "d872427b-ff64-435c-9486-ec315dfc11cb" (UID: "d872427b-ff64-435c-9486-ec315dfc11cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:47:04.539238 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.539223    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-kube-api-access-vx4ph" pod "d872427b-ff64-435c-9486-ec315dfc11cb" (UID: "d872427b-ff64-435c-9486-ec315dfc11cb"). InnerVolumeSpecName "kube-api-access-vx4ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:04.539305 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.539238    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-envoy" pod "d872427b-ff64-435c-9486-ec315dfc11cb" (UID: "d872427b-ff64-435c-9486-ec315dfc11cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:04.638582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.638551    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx4ph\" (UniqueName: \"kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-kube-api-access-vx4ph\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.638582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.638576    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.638582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.638586    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.638888 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.638595    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d872427b-ff64-435c-9486-ec315dfc11cb-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.638888 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.638603    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d872427b-ff64-435c-9486-ec315dfc11cb-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:04.638888 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:04.638611    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d872427b-ff64-435c-9486-ec315dfc11cb-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:05.199454 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:05.199421    4484 generic.go:356] "Generic (PLEG): container finished" podID="d872427b-ff64-435c-9486-ec315dfc11cb" containerID="7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a" exitCode=0
Oct 31 19:47:05.199683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:05.199464    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"
Oct 31 19:47:05.199683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:05.199495    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" event={"ID":"d872427b-ff64-435c-9486-ec315dfc11cb","Type":"ContainerDied","Data":"7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a"}
Oct 31 19:47:05.199683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:05.199551    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx" event={"ID":"d872427b-ff64-435c-9486-ec315dfc11cb","Type":"ContainerDied","Data":"05f0cadee858cddb8a6ba0a5d838c168bc17b8f5d05aa185214482625cdc4a39"}
Oct 31 19:47:05.199683 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:05.199579    4484 scope.go:122] "RemoveContainer" containerID="7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a"
Oct 31 19:47:05.209389 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:05.209370    4484 scope.go:122] "RemoveContainer" containerID="7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a"
Oct 31 19:47:05.209688 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:47:05.209666    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a\": container with ID starting with 7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a not found: ID does not exist" containerID="7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a"
Oct 31 19:47:05.209770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:05.209696    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a"} err="failed to get container status \"7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a\": rpc error: code = NotFound desc = could not find container \"7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a\": container with ID starting with 7f6dd45fedd727b0858685d0b7cd6886730e8989a7099022aa387ec00f80619a not found: ID does not exist"
Oct 31 19:47:05.372331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:05.372297    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"]
Oct 31 19:47:05.423244 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:05.423213    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-da6ea7-openshift-default-5c94dffcc9-nx4gx"]
Oct 31 19:47:05.505050 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:05.505024    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vhjgq_26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8/server/0.log"
Oct 31 19:47:05.980991 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:05.980892    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xxg6_9fc7ddc9-d304-433d-9f6a-55d677ec5ad0/proxy/0.log"
Oct 31 19:47:05.992055 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:05.992025    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" path="/var/lib/kubelet/pods/d872427b-ff64-435c-9486-ec315dfc11cb/volumes"
Oct 31 19:47:06.633638 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:06.633604    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c8b7d8cc9-rzk2k_ea33ee05-e391-4d64-a57a-039b32d83cd2/manager/0.log"
Oct 31 19:47:11.183090 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.183060    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:47:11.292836 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.292797    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"]
Oct 31 19:47:11.293168 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293156    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a77aa675-9822-44eb-9373-7a473b9896a4" containerName="cleanup"
Oct 31 19:47:11.293220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293170    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77aa675-9822-44eb-9373-7a473b9896a4" containerName="cleanup"
Oct 31 19:47:11.293220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293182    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0575483e-549c-4b63-bc9f-72218b8d72d8" containerName="maas-api"
Oct 31 19:47:11.293220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293188    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0575483e-549c-4b63-bc9f-72218b8d72d8" containerName="maas-api"
Oct 31 19:47:11.293220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293201    4484 cpu_manager.go:414] "containerMap: removing container" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" containerName="istio-proxy"
Oct 31 19:47:11.293220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293206    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" containerName="istio-proxy"
Oct 31 19:47:11.293220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293215    4484 cpu_manager.go:414] "containerMap: removing container" podUID="486fd098-2d6b-4a72-a28a-b7203b42bb61" containerName="payload-processing"
Oct 31 19:47:11.293220 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293220    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486fd098-2d6b-4a72-a28a-b7203b42bb61" containerName="payload-processing"
Oct 31 19:47:11.293492 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293236    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0ff209d0-91a6-4b4e-9fd5-fe7258c12c96" containerName="payload-pre-processing"
Oct 31 19:47:11.293492 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293240    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff209d0-91a6-4b4e-9fd5-fe7258c12c96" containerName="payload-pre-processing"
Oct 31 19:47:11.293492 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293302    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff209d0-91a6-4b4e-9fd5-fe7258c12c96" containerName="payload-pre-processing"
Oct 31 19:47:11.293492 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293320    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77aa675-9822-44eb-9373-7a473b9896a4" containerName="cleanup"
Oct 31 19:47:11.293492 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293326    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0575483e-549c-4b63-bc9f-72218b8d72d8" containerName="maas-api"
Oct 31 19:47:11.293492 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293334    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d872427b-ff64-435c-9486-ec315dfc11cb" containerName="istio-proxy"
Oct 31 19:47:11.293492 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.293341    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="486fd098-2d6b-4a72-a28a-b7203b42bb61" containerName="payload-processing"
Oct 31 19:47:11.296345 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.296326    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.352225 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.352197    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-934ace10-openshift-default-dockercfg-8jkb4\""
Oct 31 19:47:11.399977 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.399940    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-socket\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.400153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.399989    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-data\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.400153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.400124    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-podinfo\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.400236 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.400161    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-certs\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.400236 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.400188    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-token\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.400296 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.400257    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istiod-ca-cert\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.400296 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.400291    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-credential-socket\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.400364 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.400310    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-envoy\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.400364 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.400335    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlkx5\" (UniqueName: \"kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-kube-api-access-hlkx5\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.404878 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.404856    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"]
Oct 31 19:47:11.501200 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501166    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-podinfo\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.501200 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501205    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-certs\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.501503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501222    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-token\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.501503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501250    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istiod-ca-cert\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.501503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501282    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-credential-socket\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.501503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501300    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-envoy\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.501503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501351    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlkx5\" (UniqueName: \"kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-kube-api-access-hlkx5\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.501503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501393    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-socket\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.501503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501422    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-data\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.501980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501699    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-certs\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.501980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501739    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-credential-socket\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.501980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501810    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-data\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.501980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.501916    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-socket\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.502356 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.502006    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istiod-ca-cert\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.503787 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.503766    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-envoy\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.504031 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.504015    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-podinfo\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.560851 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.560817    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-token\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.560997 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.560926    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlkx5\" (UniqueName: \"kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-kube-api-access-hlkx5\") pod \"e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") " pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:11.609805 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:11.609763    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:12.032736 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:12.032704    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"]
Oct 31 19:47:12.072162 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:47:12.072126    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb65593_dfd8_4fd1_aa55_31ae22e9f5f6.slice/crio-721cc81d5ccef8b3b7f21f0c0a0a39d6504d010f74360fe3bf05ade757828778 WatchSource:0}: Error finding container 721cc81d5ccef8b3b7f21f0c0a0a39d6504d010f74360fe3bf05ade757828778: Status 404 returned error can't find the container with id 721cc81d5ccef8b3b7f21f0c0a0a39d6504d010f74360fe3bf05ade757828778
Oct 31 19:47:12.074183 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:12.074149    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:47:12.074280 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:12.074220    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:47:12.074280 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:12.074246    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:47:12.235422 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:12.235385    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" event={"ID":"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6","Type":"ContainerStarted","Data":"4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108"}
Oct 31 19:47:12.235894 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:12.235449    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" event={"ID":"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6","Type":"ContainerStarted","Data":"721cc81d5ccef8b3b7f21f0c0a0a39d6504d010f74360fe3bf05ade757828778"}
Oct 31 19:47:12.354971 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:12.354858    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" podStartSLOduration=1.354839497 podStartE2EDuration="1.354839497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:12.354185479 +0000 UTC m=+5178.500635347" watchObservedRunningTime="2026-10-31 19:47:12.354839497 +0000 UTC m=+5178.501289361"
Oct 31 19:47:12.610958 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:12.610871    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:13.615587 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:13.615557    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:14.245387 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:14.245354    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:14.246380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:14.246360    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:18.000200 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.000169    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"]
Oct 31 19:47:18.003784 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.003764    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"
Oct 31 19:47:18.061072 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.061037    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"]
Oct 31 19:47:18.109736 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.109703    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-934ace10-dockercfg-5x2cz\""
Oct 31 19:47:18.109928 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.109756    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-934ace10\""
Oct 31 19:47:18.162610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.162578    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"]
Oct 31 19:47:18.166136 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.166112    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt\" (UID: \"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"
Oct 31 19:47:18.166267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.166143    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"
Oct 31 19:47:18.166267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.166144    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf5f4\" (UniqueName: \"kubernetes.io/projected/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-kube-api-access-jf5f4\") pod \"payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt\" (UID: \"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"
Oct 31 19:47:18.209932 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.209903    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"]
Oct 31 19:47:18.267369 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.267296    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd23a2b-1afa-4674-879d-d633bfee0ff9-plugins-config-volume\") pod \"payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct\" (UID: \"1fd23a2b-1afa-4674-879d-d633bfee0ff9\") " pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"
Oct 31 19:47:18.267536 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.267376    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7hj\" (UniqueName: \"kubernetes.io/projected/1fd23a2b-1afa-4674-879d-d633bfee0ff9-kube-api-access-wb7hj\") pod \"payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct\" (UID: \"1fd23a2b-1afa-4674-879d-d633bfee0ff9\") " pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"
Oct 31 19:47:18.267536 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.267440    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt\" (UID: \"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"
Oct 31 19:47:18.267536 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.267462    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf5f4\" (UniqueName: \"kubernetes.io/projected/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-kube-api-access-jf5f4\") pod \"payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt\" (UID: \"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"
Oct 31 19:47:18.268182 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.268164    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt\" (UID: \"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"
Oct 31 19:47:18.325783 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.325754    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf5f4\" (UniqueName: \"kubernetes.io/projected/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-kube-api-access-jf5f4\") pod \"payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt\" (UID: \"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"
Oct 31 19:47:18.368626 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.368599    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7hj\" (UniqueName: \"kubernetes.io/projected/1fd23a2b-1afa-4674-879d-d633bfee0ff9-kube-api-access-wb7hj\") pod \"payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct\" (UID: \"1fd23a2b-1afa-4674-879d-d633bfee0ff9\") " pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"
Oct 31 19:47:18.368774 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.368674    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd23a2b-1afa-4674-879d-d633bfee0ff9-plugins-config-volume\") pod \"payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct\" (UID: \"1fd23a2b-1afa-4674-879d-d633bfee0ff9\") " pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"
Oct 31 19:47:18.369338 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.369319    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd23a2b-1afa-4674-879d-d633bfee0ff9-plugins-config-volume\") pod \"payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct\" (UID: \"1fd23a2b-1afa-4674-879d-d633bfee0ff9\") " pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"
Oct 31 19:47:18.427293 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.427265    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7hj\" (UniqueName: \"kubernetes.io/projected/1fd23a2b-1afa-4674-879d-d633bfee0ff9-kube-api-access-wb7hj\") pod \"payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct\" (UID: \"1fd23a2b-1afa-4674-879d-d633bfee0ff9\") " pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"
Oct 31 19:47:18.478575 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.478550    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"
Oct 31 19:47:18.616653 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.616615    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"
Oct 31 19:47:18.846505 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.846458    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"]
Oct 31 19:47:18.992310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:18.992283    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"]
Oct 31 19:47:18.993665 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:47:18.993636    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1faba4e_f8c2_40e6_88f4_27dcb06e5f59.slice/crio-318086457ac03f6c8ea7ee2d9445b3bdfac80feb02b2204e00562182665a1cb1 WatchSource:0}: Error finding container 318086457ac03f6c8ea7ee2d9445b3bdfac80feb02b2204e00562182665a1cb1: Status 404 returned error can't find the container with id 318086457ac03f6c8ea7ee2d9445b3bdfac80feb02b2204e00562182665a1cb1
Oct 31 19:47:19.271119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:19.271079    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt" event={"ID":"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59","Type":"ContainerStarted","Data":"5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638"}
Oct 31 19:47:19.271657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:19.271133    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt" event={"ID":"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59","Type":"ContainerStarted","Data":"318086457ac03f6c8ea7ee2d9445b3bdfac80feb02b2204e00562182665a1cb1"}
Oct 31 19:47:19.271657 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:19.271262    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"
Oct 31 19:47:19.272765 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:19.272740    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct" event={"ID":"1fd23a2b-1afa-4674-879d-d633bfee0ff9","Type":"ContainerStarted","Data":"1f0c93f605975e4a30e9eb714bb920ab945d864342893cf606975e47ea0bb15a"}
Oct 31 19:47:19.272864 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:19.272790    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"
Oct 31 19:47:19.272864 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:19.272809    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct" event={"ID":"1fd23a2b-1afa-4674-879d-d633bfee0ff9","Type":"ContainerStarted","Data":"10af86a61228067c88d79e9f249ff8bf6a77aac55cb3aef481d3a2a01abbd66f"}
Oct 31 19:47:19.389653 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:19.389595    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt" podStartSLOduration=2.389581404 podStartE2EDuration="2.389581404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:19.38693047 +0000 UTC m=+5185.533380326" watchObservedRunningTime="2026-10-31 19:47:19.389581404 +0000 UTC m=+5185.536031267"
Oct 31 19:47:19.504800 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:19.504741    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct" podStartSLOduration=1.504727415 podStartE2EDuration="1.504727415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:19.503902878 +0000 UTC m=+5185.650352742" watchObservedRunningTime="2026-10-31 19:47:19.504727415 +0000 UTC m=+5185.651177279"
Oct 31 19:47:25.467973 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:47:25.467929    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:47:28.363856 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:28.363826    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:47:30.286545 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:30.286516    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"
Oct 31 19:47:30.287211 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:30.287190    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"
Oct 31 19:47:38.368087 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:38.368050    4484 generic.go:356] "Generic (PLEG): container finished" podID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerID="4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108" exitCode=0
Oct 31 19:47:38.368087 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:38.368084    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" event={"ID":"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6","Type":"ContainerDied","Data":"4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108"}
Oct 31 19:47:38.368779 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:38.368508    4484 scope.go:122] "RemoveContainer" containerID="4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108"
Oct 31 19:47:38.369855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:38.369828    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:47:38.369929 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:38.369897    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:47:38.369965 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:38.369931    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:47:39.391006 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:39.390973    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" event={"ID":"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6","Type":"ContainerStarted","Data":"fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6"}
Oct 31 19:47:39.610331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:39.610298    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:39.611485 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:39.611448    4484 patch_prober.go:29] interesting pod/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Oct 31 19:47:39.611610 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:39.611515    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Oct 31 19:47:40.611138 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:40.611107    4484 patch_prober.go:29] interesting pod/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Oct 31 19:47:40.611565 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:40.611159    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Oct 31 19:47:41.610942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:41.610903    4484 patch_prober.go:29] interesting pod/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Oct 31 19:47:41.611128 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:41.610955    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Oct 31 19:47:42.610700 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:42.610669    4484 patch_prober.go:29] interesting pod/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Oct 31 19:47:42.611102 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:42.610726    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Oct 31 19:47:43.610617 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:43.610585    4484 patch_prober.go:29] interesting pod/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Oct 31 19:47:43.610807 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:43.610637    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Oct 31 19:47:44.245781 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:44.245736    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:44.615006 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:44.614930    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:44.616173 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:44.616146    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:47:50.872048 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:50.872016    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"]
Oct 31 19:47:50.875758 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:50.875739    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:50.932354 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:50.932319    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"]
Oct 31 19:47:50.980666 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:50.980636    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:47:50.980828 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:50.980748    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:47:50.995876 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:50.995849    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-934ace10-n5lq2\" (UID: \"739a2245-9c1e-4b71-887b-ed09b8d6767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:50.996043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:50.995971    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mrh\" (UniqueName: \"kubernetes.io/projected/739a2245-9c1e-4b71-887b-ed09b8d6767d-kube-api-access-g6mrh\") pod \"maas-api-revoke-keys-e2e-ait-934ace10-n5lq2\" (UID: \"739a2245-9c1e-4b71-887b-ed09b8d6767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:50.996043 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:50.996008    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-934ace10-n5lq2\" (UID: \"739a2245-9c1e-4b71-887b-ed09b8d6767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:51.097453 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:51.097422    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mrh\" (UniqueName: \"kubernetes.io/projected/739a2245-9c1e-4b71-887b-ed09b8d6767d-kube-api-access-g6mrh\") pod \"maas-api-revoke-keys-e2e-ait-934ace10-n5lq2\" (UID: \"739a2245-9c1e-4b71-887b-ed09b8d6767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:51.097453 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:51.097457    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-934ace10-n5lq2\" (UID: \"739a2245-9c1e-4b71-887b-ed09b8d6767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:51.097719 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:51.097508    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-934ace10-n5lq2\" (UID: \"739a2245-9c1e-4b71-887b-ed09b8d6767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:51.098145 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:51.098124    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-934ace10-n5lq2\" (UID: \"739a2245-9c1e-4b71-887b-ed09b8d6767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:51.100069 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:51.100040    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-934ace10-n5lq2\" (UID: \"739a2245-9c1e-4b71-887b-ed09b8d6767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:51.156346 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:51.156279    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mrh\" (UniqueName: \"kubernetes.io/projected/739a2245-9c1e-4b71-887b-ed09b8d6767d-kube-api-access-g6mrh\") pod \"maas-api-revoke-keys-e2e-ait-934ace10-n5lq2\" (UID: \"739a2245-9c1e-4b71-887b-ed09b8d6767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:51.189120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:51.189093    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:51.570377 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:51.570352    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"]
Oct 31 19:47:51.572313 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:47:51.572288    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739a2245_9c1e_4b71_887b_ed09b8d6767d.slice/crio-4245ed3039b7048313b2f1cb150673ddadd86259b1eb8330b537d7d3f720d872 WatchSource:0}: Error finding container 4245ed3039b7048313b2f1cb150673ddadd86259b1eb8330b537d7d3f720d872: Status 404 returned error can't find the container with id 4245ed3039b7048313b2f1cb150673ddadd86259b1eb8330b537d7d3f720d872
Oct 31 19:47:52.456082 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:52.456044    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2" event={"ID":"739a2245-9c1e-4b71-887b-ed09b8d6767d","Type":"ContainerStarted","Data":"b1f7ee859896d5e0b5b2c9c69fdc49481849e63956e6b89565170e41f43367b7"}
Oct 31 19:47:52.456505 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:52.456101    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2" event={"ID":"739a2245-9c1e-4b71-887b-ed09b8d6767d","Type":"ContainerStarted","Data":"4245ed3039b7048313b2f1cb150673ddadd86259b1eb8330b537d7d3f720d872"}
Oct 31 19:47:52.573828 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:52.573784    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2" podStartSLOduration=2.5737713639999997 podStartE2EDuration="2.573771364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:52.573412703 +0000 UTC m=+5218.719862565" watchObservedRunningTime="2026-10-31 19:47:52.573771364 +0000 UTC m=+5218.720221231"
Oct 31 19:47:53.463186 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:53.463155    4484 generic.go:356] "Generic (PLEG): container finished" podID="739a2245-9c1e-4b71-887b-ed09b8d6767d" containerID="b1f7ee859896d5e0b5b2c9c69fdc49481849e63956e6b89565170e41f43367b7" exitCode=0
Oct 31 19:47:53.463614 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:53.463227    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2" event={"ID":"739a2245-9c1e-4b71-887b-ed09b8d6767d","Type":"ContainerDied","Data":"b1f7ee859896d5e0b5b2c9c69fdc49481849e63956e6b89565170e41f43367b7"}
Oct 31 19:47:54.595117 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:54.595091    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:54.731140 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:54.731060    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739a2245-9c1e-4b71-887b-ed09b8d6767d-kube-api-access-g6mrh\" (UniqueName: \"kubernetes.io/projected/739a2245-9c1e-4b71-887b-ed09b8d6767d-kube-api-access-g6mrh\") pod \"739a2245-9c1e-4b71-887b-ed09b8d6767d\" (UID: \"739a2245-9c1e-4b71-887b-ed09b8d6767d\") "
Oct 31 19:47:54.731140 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:54.731123    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-openshift-ca\") pod \"739a2245-9c1e-4b71-887b-ed09b8d6767d\" (UID: \"739a2245-9c1e-4b71-887b-ed09b8d6767d\") "
Oct 31 19:47:54.731384 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:54.731152    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-xks-ca\") pod \"739a2245-9c1e-4b71-887b-ed09b8d6767d\" (UID: \"739a2245-9c1e-4b71-887b-ed09b8d6767d\") "
Oct 31 19:47:54.731525 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:54.731498    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-openshift-ca" pod "739a2245-9c1e-4b71-887b-ed09b8d6767d" (UID: "739a2245-9c1e-4b71-887b-ed09b8d6767d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:54.733325 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:54.733287    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739a2245-9c1e-4b71-887b-ed09b8d6767d-kube-api-access-g6mrh" pod "739a2245-9c1e-4b71-887b-ed09b8d6767d" (UID: "739a2245-9c1e-4b71-887b-ed09b8d6767d"). InnerVolumeSpecName "kube-api-access-g6mrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:54.733455 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:54.733358    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-xks-ca" pod "739a2245-9c1e-4b71-887b-ed09b8d6767d" (UID: "739a2245-9c1e-4b71-887b-ed09b8d6767d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:47:54.832327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:54.832297    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6mrh\" (UniqueName: \"kubernetes.io/projected/739a2245-9c1e-4b71-887b-ed09b8d6767d-kube-api-access-g6mrh\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:54.832327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:54.832323    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:54.832327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:54.832333    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739a2245-9c1e-4b71-887b-ed09b8d6767d-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:47:55.466856 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:47:55.466821    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:47:55.475627 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:55.475604    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"
Oct 31 19:47:55.475792 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:55.475600    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2" event={"ID":"739a2245-9c1e-4b71-887b-ed09b8d6767d","Type":"ContainerDied","Data":"4245ed3039b7048313b2f1cb150673ddadd86259b1eb8330b537d7d3f720d872"}
Oct 31 19:47:55.475792 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:55.475719    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4245ed3039b7048313b2f1cb150673ddadd86259b1eb8330b537d7d3f720d872"
Oct 31 19:47:56.035094 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:47:56.035057    4484 scope.go:122] "RemoveContainer" containerID="44ebdaae49f6bc9796154303ec6fbf573833ac37684099b92e0942ad3e9f3f11"
Oct 31 19:48:02.346510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:02.346457    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"]
Oct 31 19:48:02.453378 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:02.453341    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"]
Oct 31 19:48:07.347231 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.347177    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct" podUID="1fd23a2b-1afa-4674-879d-d633bfee0ff9" containerName="payload-processing" containerID="cri-o://1f0c93f605975e4a30e9eb714bb920ab945d864342893cf606975e47ea0bb15a" gracePeriod=25
Oct 31 19:48:07.454115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.454079    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt" podUID="a1faba4e-f8c2-40e6-88f4-27dcb06e5f59" containerName="payload-pre-processing" containerID="cri-o://5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638" gracePeriod=25
Oct 31 19:48:07.544463 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.544428    4484 generic.go:356] "Generic (PLEG): container finished" podID="1fd23a2b-1afa-4674-879d-d633bfee0ff9" containerID="1f0c93f605975e4a30e9eb714bb920ab945d864342893cf606975e47ea0bb15a" exitCode=0
Oct 31 19:48:07.544648 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.544466    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct" event={"ID":"1fd23a2b-1afa-4674-879d-d633bfee0ff9","Type":"ContainerDied","Data":"1f0c93f605975e4a30e9eb714bb920ab945d864342893cf606975e47ea0bb15a"}
Oct 31 19:48:07.601724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.601659    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"
Oct 31 19:48:07.702120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.702098    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"
Oct 31 19:48:07.750589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.750565    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd23a2b-1afa-4674-879d-d633bfee0ff9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd23a2b-1afa-4674-879d-d633bfee0ff9-plugins-config-volume\") pod \"1fd23a2b-1afa-4674-879d-d633bfee0ff9\" (UID: \"1fd23a2b-1afa-4674-879d-d633bfee0ff9\") "
Oct 31 19:48:07.750758 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.750596    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd23a2b-1afa-4674-879d-d633bfee0ff9-kube-api-access-wb7hj\" (UniqueName: \"kubernetes.io/projected/1fd23a2b-1afa-4674-879d-d633bfee0ff9-kube-api-access-wb7hj\") pod \"1fd23a2b-1afa-4674-879d-d633bfee0ff9\" (UID: \"1fd23a2b-1afa-4674-879d-d633bfee0ff9\") "
Oct 31 19:48:07.750917 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.750897    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd23a2b-1afa-4674-879d-d633bfee0ff9-plugins-config-volume" pod "1fd23a2b-1afa-4674-879d-d633bfee0ff9" (UID: "1fd23a2b-1afa-4674-879d-d633bfee0ff9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:07.752861 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.752834    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd23a2b-1afa-4674-879d-d633bfee0ff9-kube-api-access-wb7hj" pod "1fd23a2b-1afa-4674-879d-d633bfee0ff9" (UID: "1fd23a2b-1afa-4674-879d-d633bfee0ff9"). InnerVolumeSpecName "kube-api-access-wb7hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:07.851103 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.851070    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-plugins-config-volume\") pod \"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59\" (UID: \"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59\") "
Oct 31 19:48:07.851298 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.851137    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-kube-api-access-jf5f4\" (UniqueName: \"kubernetes.io/projected/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-kube-api-access-jf5f4\") pod \"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59\" (UID: \"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59\") "
Oct 31 19:48:07.851414 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.851392    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-plugins-config-volume" pod "a1faba4e-f8c2-40e6-88f4-27dcb06e5f59" (UID: "a1faba4e-f8c2-40e6-88f4-27dcb06e5f59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:07.851510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.851463    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd23a2b-1afa-4674-879d-d633bfee0ff9-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:07.851510 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.851501    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7hj\" (UniqueName: \"kubernetes.io/projected/1fd23a2b-1afa-4674-879d-d633bfee0ff9-kube-api-access-wb7hj\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:07.853385 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.853324    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-kube-api-access-jf5f4" pod "a1faba4e-f8c2-40e6-88f4-27dcb06e5f59" (UID: "a1faba4e-f8c2-40e6-88f4-27dcb06e5f59"). InnerVolumeSpecName "kube-api-access-jf5f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:07.952902 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.952870    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:07.952902 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:07.952895    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf5f4\" (UniqueName: \"kubernetes.io/projected/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59-kube-api-access-jf5f4\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:08.558341 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.558308    4484 generic.go:356] "Generic (PLEG): container finished" podID="a1faba4e-f8c2-40e6-88f4-27dcb06e5f59" containerID="5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638" exitCode=0
Oct 31 19:48:08.558840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.558362    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"
Oct 31 19:48:08.558840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.558392    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt" event={"ID":"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59","Type":"ContainerDied","Data":"5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638"}
Oct 31 19:48:08.558840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.558493    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt" event={"ID":"a1faba4e-f8c2-40e6-88f4-27dcb06e5f59","Type":"ContainerDied","Data":"318086457ac03f6c8ea7ee2d9445b3bdfac80feb02b2204e00562182665a1cb1"}
Oct 31 19:48:08.558840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.558522    4484 scope.go:122] "RemoveContainer" containerID="5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638"
Oct 31 19:48:08.560151 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.560113    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct" event={"ID":"1fd23a2b-1afa-4674-879d-d633bfee0ff9","Type":"ContainerDied","Data":"10af86a61228067c88d79e9f249ff8bf6a77aac55cb3aef481d3a2a01abbd66f"}
Oct 31 19:48:08.560313 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.560290    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"
Oct 31 19:48:08.568165 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.568149    4484 scope.go:122] "RemoveContainer" containerID="5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638"
Oct 31 19:48:08.568464 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:48:08.568441    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638\": container with ID starting with 5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638 not found: ID does not exist" containerID="5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638"
Oct 31 19:48:08.568614 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.568486    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638"} err="failed to get container status \"5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638\": rpc error: code = NotFound desc = could not find container \"5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638\": container with ID starting with 5a3b44de811cb7b39bd3b0aea7da52f83285be765244bcc1dcd9e01347236638 not found: ID does not exist"
Oct 31 19:48:08.568614 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.568509    4484 scope.go:122] "RemoveContainer" containerID="1f0c93f605975e4a30e9eb714bb920ab945d864342893cf606975e47ea0bb15a"
Oct 31 19:48:08.727114 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.727024    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"]
Oct 31 19:48:08.777696 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.777662    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-934ace10-764f7bcc75-d89lt"]
Oct 31 19:48:08.888387 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.888352    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"]
Oct 31 19:48:08.894140 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:08.894114    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-934ace10-5f8bbf5855-z86ct"]
Oct 31 19:48:09.991725 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:09.991690    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd23a2b-1afa-4674-879d-d633bfee0ff9" path="/var/lib/kubelet/pods/1fd23a2b-1afa-4674-879d-d633bfee0ff9/volumes"
Oct 31 19:48:09.992073 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:09.992056    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1faba4e-f8c2-40e6-88f4-27dcb06e5f59" path="/var/lib/kubelet/pods/a1faba4e-f8c2-40e6-88f4-27dcb06e5f59/volumes"
Oct 31 19:48:10.209691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:10.209654    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"]
Oct 31 19:48:10.260445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:10.260359    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-934ace10-n5lq2"]
Oct 31 19:48:11.992882 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:11.992851    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739a2245-9c1e-4b71-887b-ed09b8d6767d" path="/var/lib/kubelet/pods/739a2245-9c1e-4b71-887b-ed09b8d6767d/volumes"
Oct 31 19:48:11.993273 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:11.993187    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"]
Oct 31 19:48:11.993461 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:11.993437    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy" containerID="cri-o://fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6" gracePeriod=30
Oct 31 19:48:12.647702 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:12.647665    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:48:14.618134 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:14.618095    4484 patch_prober.go:29] interesting pod/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:48:14.618542 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:14.618169    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:48:17.246245 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.246215    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:48:17.440871 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.440771    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-certs\") pod \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") "
Oct 31 19:48:17.440871 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.440824    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-kube-api-access-hlkx5\" (UniqueName: \"kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-kube-api-access-hlkx5\") pod \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") "
Oct 31 19:48:17.440871 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.440853    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-envoy\") pod \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") "
Oct 31 19:48:17.441185 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.440933    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-credential-socket\") pod \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") "
Oct 31 19:48:17.441185 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.440981    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-podinfo\") pod \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") "
Oct 31 19:48:17.441185 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.441015    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-socket\") pod \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") "
Oct 31 19:48:17.441185 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.441046    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-data\") pod \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") "
Oct 31 19:48:17.441185 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.441090    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-token\" (UniqueName: \"kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-token\") pod \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") "
Oct 31 19:48:17.441185 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.441095    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-certs" pod "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" (UID: "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:17.441185 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.441161    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istiod-ca-cert\") pod \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\" (UID: \"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6\") "
Oct 31 19:48:17.441651 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.441417    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:17.441651 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.441529    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-credential-socket" pod "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" (UID: "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:17.441770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.441727    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-data" pod "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" (UID: "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:17.441770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.441757    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istiod-ca-cert" pod "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" (UID: "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:17.442177 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.442081    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-socket" pod "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" (UID: "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:17.444281 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.444258    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-podinfo" pod "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" (UID: "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:48:17.444445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.444413    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-kube-api-access-hlkx5" pod "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" (UID: "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6"). InnerVolumeSpecName "kube-api-access-hlkx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:17.444803 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.444778    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-token" pod "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" (UID: "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:17.444889 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.444841    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-envoy" pod "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" (UID: "cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:17.542392 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.542353    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:17.542392 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.542386    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:17.542392 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.542399    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlkx5\" (UniqueName: \"kubernetes.io/projected/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-kube-api-access-hlkx5\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:17.542673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.542411    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:17.542673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.542425    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:17.542673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.542436    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:17.542673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.542447    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:17.542673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.542458    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:17.606809 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.606778    4484 generic.go:356] "Generic (PLEG): container finished" podID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerID="fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6" exitCode=0
Oct 31 19:48:17.607004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.606832    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"
Oct 31 19:48:17.607004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.606839    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" event={"ID":"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6","Type":"ContainerDied","Data":"fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6"}
Oct 31 19:48:17.607004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.606976    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7" event={"ID":"cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6","Type":"ContainerDied","Data":"721cc81d5ccef8b3b7f21f0c0a0a39d6504d010f74360fe3bf05ade757828778"}
Oct 31 19:48:17.607169 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.607018    4484 scope.go:122] "RemoveContainer" containerID="fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6"
Oct 31 19:48:17.616912 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.616885    4484 scope.go:122] "RemoveContainer" containerID="4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108"
Oct 31 19:48:17.625609 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.625588    4484 scope.go:122] "RemoveContainer" containerID="fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6"
Oct 31 19:48:17.625914 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:48:17.625895    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6\": container with ID starting with fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6 not found: ID does not exist" containerID="fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6"
Oct 31 19:48:17.625977 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.625922    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6"} err="failed to get container status \"fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6\": rpc error: code = NotFound desc = could not find container \"fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6\": container with ID starting with fa1313e7938560a66ca520230635b72ff42f64f59882b83c31cc9b78398485a6 not found: ID does not exist"
Oct 31 19:48:17.625977 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.625938    4484 scope.go:122] "RemoveContainer" containerID="4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108"
Oct 31 19:48:17.626203 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:48:17.626187    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108\": container with ID starting with 4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108 not found: ID does not exist" containerID="4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108"
Oct 31 19:48:17.626249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.626207    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108"} err="failed to get container status \"4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108\": rpc error: code = NotFound desc = could not find container \"4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108\": container with ID starting with 4fd1d763fa19217bf332d52c0bd1cc8126f7aeea8648dbc0463f2898bddd3108 not found: ID does not exist"
Oct 31 19:48:17.774712 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.774663    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"]
Oct 31 19:48:17.779609 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.779568    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-934ace10-openshift-default-7c8b97ccb5-5dfz7"]
Oct 31 19:48:17.994668 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:17.994635    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" path="/var/lib/kubelet/pods/cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6/volumes"
Oct 31 19:48:19.639523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.639493    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-7zfzw"]
Oct 31 19:48:19.639892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.639860    4484 cpu_manager.go:414] "containerMap: removing container" podUID="1fd23a2b-1afa-4674-879d-d633bfee0ff9" containerName="payload-processing"
Oct 31 19:48:19.639892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.639871    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd23a2b-1afa-4674-879d-d633bfee0ff9" containerName="payload-processing"
Oct 31 19:48:19.639892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.639885    4484 cpu_manager.go:414] "containerMap: removing container" podUID="739a2245-9c1e-4b71-887b-ed09b8d6767d" containerName="revoke-keys"
Oct 31 19:48:19.639892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.639890    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739a2245-9c1e-4b71-887b-ed09b8d6767d" containerName="revoke-keys"
Oct 31 19:48:19.640024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.639907    4484 cpu_manager.go:414] "containerMap: removing container" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy"
Oct 31 19:48:19.640024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.639912    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy"
Oct 31 19:48:19.640024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.639923    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a1faba4e-f8c2-40e6-88f4-27dcb06e5f59" containerName="payload-pre-processing"
Oct 31 19:48:19.640024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.639927    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1faba4e-f8c2-40e6-88f4-27dcb06e5f59" containerName="payload-pre-processing"
Oct 31 19:48:19.640024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.639935    4484 cpu_manager.go:414] "containerMap: removing container" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy"
Oct 31 19:48:19.640024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.639940    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy"
Oct 31 19:48:19.640024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.640000    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy"
Oct 31 19:48:19.640024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.640009    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd23a2b-1afa-4674-879d-d633bfee0ff9" containerName="payload-processing"
Oct 31 19:48:19.640024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.640016    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="739a2245-9c1e-4b71-887b-ed09b8d6767d" containerName="revoke-keys"
Oct 31 19:48:19.640024 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.640025    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1faba4e-f8c2-40e6-88f4-27dcb06e5f59" containerName="payload-pre-processing"
Oct 31 19:48:19.644521 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.644501    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:19.697893 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.697869    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 19:48:19.748885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.748860    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 19:48:19.749035 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.748913    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 19:48:19.749035 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.748957    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-clhwj\""
Oct 31 19:48:19.752251 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.752227    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-7zfzw"]
Oct 31 19:48:19.759379 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.759358    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-webhook-cert\") pod \"maas-controller-64fbfd7b7f-7zfzw\" (UID: \"1572ca38-17ef-41d9-b56a-a04bb167c025\") " pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:19.759459 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.759391    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-metrics-certs\") pod \"maas-controller-64fbfd7b7f-7zfzw\" (UID: \"1572ca38-17ef-41d9-b56a-a04bb167c025\") " pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:19.759459 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.759419    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx5vb\" (UniqueName: \"kubernetes.io/projected/1572ca38-17ef-41d9-b56a-a04bb167c025-kube-api-access-jx5vb\") pod \"maas-controller-64fbfd7b7f-7zfzw\" (UID: \"1572ca38-17ef-41d9-b56a-a04bb167c025\") " pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:19.860231 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.860202    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-webhook-cert\") pod \"maas-controller-64fbfd7b7f-7zfzw\" (UID: \"1572ca38-17ef-41d9-b56a-a04bb167c025\") " pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:19.860406 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.860244    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-metrics-certs\") pod \"maas-controller-64fbfd7b7f-7zfzw\" (UID: \"1572ca38-17ef-41d9-b56a-a04bb167c025\") " pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:19.860406 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.860283    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx5vb\" (UniqueName: \"kubernetes.io/projected/1572ca38-17ef-41d9-b56a-a04bb167c025-kube-api-access-jx5vb\") pod \"maas-controller-64fbfd7b7f-7zfzw\" (UID: \"1572ca38-17ef-41d9-b56a-a04bb167c025\") " pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:19.862844 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.862816    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-metrics-certs\") pod \"maas-controller-64fbfd7b7f-7zfzw\" (UID: \"1572ca38-17ef-41d9-b56a-a04bb167c025\") " pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:19.862962 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.862870    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-webhook-cert\") pod \"maas-controller-64fbfd7b7f-7zfzw\" (UID: \"1572ca38-17ef-41d9-b56a-a04bb167c025\") " pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:19.919278 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.919217    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx5vb\" (UniqueName: \"kubernetes.io/projected/1572ca38-17ef-41d9-b56a-a04bb167c025-kube-api-access-jx5vb\") pod \"maas-controller-64fbfd7b7f-7zfzw\" (UID: \"1572ca38-17ef-41d9-b56a-a04bb167c025\") " pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:19.956397 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:19.956370    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:20.330428 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:20.330399    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-7zfzw"]
Oct 31 19:48:20.331907 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:48:20.331880    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1572ca38_17ef_41d9_b56a_a04bb167c025.slice/crio-c710a5e2632e3156c2313c5980149d6a84f5990adc00762477543d6b9bdaeb3a WatchSource:0}: Error finding container c710a5e2632e3156c2313c5980149d6a84f5990adc00762477543d6b9bdaeb3a: Status 404 returned error can't find the container with id c710a5e2632e3156c2313c5980149d6a84f5990adc00762477543d6b9bdaeb3a
Oct 31 19:48:20.333190 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:20.333173    4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:48:20.624956 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:20.624872    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw" event={"ID":"1572ca38-17ef-41d9-b56a-a04bb167c025","Type":"ContainerStarted","Data":"c710a5e2632e3156c2313c5980149d6a84f5990adc00762477543d6b9bdaeb3a"}
Oct 31 19:48:22.636669 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:22.636641    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw" event={"ID":"1572ca38-17ef-41d9-b56a-a04bb167c025","Type":"ContainerStarted","Data":"7532e97c247706eb1ad257d9b5fad3471d511be1b98f386a3b2ee5bdc5906191"}
Oct 31 19:48:22.636977 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:22.636747    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:22.803870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:22.803805    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw" podStartSLOduration=1.561230903 podStartE2EDuration="3.803792304s" totalImagesPullingTime="2.242561401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:22.801759985 +0000 UTC m=+5248.948209864" watchObservedRunningTime="2026-10-31 19:48:22.803792304 +0000 UTC m=+5248.950242167"
Oct 31 19:48:25.467459 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:48:25.467425    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:48:33.643516 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:33.643462    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:43.963410 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:43.963203    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"]
Oct 31 19:48:43.964384 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:43.964354    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb65593-dfd8-4fd1-aa55-31ae22e9f5f6" containerName="istio-proxy"
Oct 31 19:48:43.974566 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:43.974537    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:44.022310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.022279    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"]
Oct 31 19:48:44.071117 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.071079    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:48:44.071283 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.071132    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:48:44.075590 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.075568    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-kw2wr\" (UID: \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:44.075684 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.075626    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-kw2wr\" (UID: \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:44.075775 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.075750    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbqpd\" (UniqueName: \"kubernetes.io/projected/a277764d-6c36-4e11-8b77-5d39ba95c0c1-kube-api-access-rbqpd\") pod \"maas-api-revoke-sub-85237e45-kw2wr\" (UID: \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:44.176443 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.176411    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-kw2wr\" (UID: \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:44.176648 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.176462    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqpd\" (UniqueName: \"kubernetes.io/projected/a277764d-6c36-4e11-8b77-5d39ba95c0c1-kube-api-access-rbqpd\") pod \"maas-api-revoke-sub-85237e45-kw2wr\" (UID: \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:44.176648 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.176597    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-kw2wr\" (UID: \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:44.177315 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.177284    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-kw2wr\" (UID: \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:44.178990 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.178964    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-kw2wr\" (UID: \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:44.234937 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.234870    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbqpd\" (UniqueName: \"kubernetes.io/projected/a277764d-6c36-4e11-8b77-5d39ba95c0c1-kube-api-access-rbqpd\") pod \"maas-api-revoke-sub-85237e45-kw2wr\" (UID: \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:44.288112 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.288079    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:44.657796 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.657765    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"]
Oct 31 19:48:44.659689 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:48:44.659651    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda277764d_6c36_4e11_8b77_5d39ba95c0c1.slice/crio-0675d3a8a7a7f2650df606c1ae0278e2b82277fbe342f1ff6fe632dc2460cc38 WatchSource:0}: Error finding container 0675d3a8a7a7f2650df606c1ae0278e2b82277fbe342f1ff6fe632dc2460cc38: Status 404 returned error can't find the container with id 0675d3a8a7a7f2650df606c1ae0278e2b82277fbe342f1ff6fe632dc2460cc38
Oct 31 19:48:44.740421 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:44.740391    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr" event={"ID":"a277764d-6c36-4e11-8b77-5d39ba95c0c1","Type":"ContainerStarted","Data":"0675d3a8a7a7f2650df606c1ae0278e2b82277fbe342f1ff6fe632dc2460cc38"}
Oct 31 19:48:45.747089 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:45.747058    4484 generic.go:356] "Generic (PLEG): container finished" podID="a277764d-6c36-4e11-8b77-5d39ba95c0c1" containerID="96c16528f50b8555493d84ce4d203a9db35ca32484597e776e7370192c1c6e94" exitCode=0
Oct 31 19:48:45.747509 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:45.747147    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr" event={"ID":"a277764d-6c36-4e11-8b77-5d39ba95c0c1","Type":"ContainerDied","Data":"96c16528f50b8555493d84ce4d203a9db35ca32484597e776e7370192c1c6e94"}
Oct 31 19:48:46.882977 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:46.882954    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:47.002380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:47.001622    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-xks-ca\") pod \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\" (UID: \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\") "
Oct 31 19:48:47.002380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:47.001697    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-openshift-ca\") pod \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\" (UID: \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\") "
Oct 31 19:48:47.002380 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:47.001750    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a277764d-6c36-4e11-8b77-5d39ba95c0c1-kube-api-access-rbqpd\" (UniqueName: \"kubernetes.io/projected/a277764d-6c36-4e11-8b77-5d39ba95c0c1-kube-api-access-rbqpd\") pod \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\" (UID: \"a277764d-6c36-4e11-8b77-5d39ba95c0c1\") "
Oct 31 19:48:47.003188 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:47.003127    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-openshift-ca" pod "a277764d-6c36-4e11-8b77-5d39ba95c0c1" (UID: "a277764d-6c36-4e11-8b77-5d39ba95c0c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:47.004636 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:47.004606    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a277764d-6c36-4e11-8b77-5d39ba95c0c1-kube-api-access-rbqpd" pod "a277764d-6c36-4e11-8b77-5d39ba95c0c1" (UID: "a277764d-6c36-4e11-8b77-5d39ba95c0c1"). InnerVolumeSpecName "kube-api-access-rbqpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:47.005353 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:47.005325    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-xks-ca" pod "a277764d-6c36-4e11-8b77-5d39ba95c0c1" (UID: "a277764d-6c36-4e11-8b77-5d39ba95c0c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:47.103158 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:47.103125    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:47.103158 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:47.103161    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a277764d-6c36-4e11-8b77-5d39ba95c0c1-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:47.103411 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:47.103177    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbqpd\" (UniqueName: \"kubernetes.io/projected/a277764d-6c36-4e11-8b77-5d39ba95c0c1-kube-api-access-rbqpd\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:47.758608 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:47.758572    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr" event={"ID":"a277764d-6c36-4e11-8b77-5d39ba95c0c1","Type":"ContainerDied","Data":"0675d3a8a7a7f2650df606c1ae0278e2b82277fbe342f1ff6fe632dc2460cc38"}
Oct 31 19:48:47.758769 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:47.758624    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0675d3a8a7a7f2650df606c1ae0278e2b82277fbe342f1ff6fe632dc2460cc38"
Oct 31 19:48:47.758769 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:47.758582    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"
Oct 31 19:48:55.046364 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:55.046326    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"]
Oct 31 19:48:55.097299 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:55.097260    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acafd54d-xh4p6"]
Oct 31 19:48:55.467346 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:48:55.467267    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:48:55.992275 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:55.992241    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67bd72e-4248-47c8-9b99-7e2393bea074" path="/var/lib/kubelet/pods/a67bd72e-4248-47c8-9b99-7e2393bea074/volumes"
Oct 31 19:48:56.645324 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:56.645290    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-7zfzw"]
Oct 31 19:48:56.645853 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:56.645536    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw" podUID="1572ca38-17ef-41d9-b56a-a04bb167c025" containerName="manager" containerID="cri-o://7532e97c247706eb1ad257d9b5fad3471d511be1b98f386a3b2ee5bdc5906191" gracePeriod=30
Oct 31 19:48:56.804005 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:56.803977    4484 generic.go:356] "Generic (PLEG): container finished" podID="1572ca38-17ef-41d9-b56a-a04bb167c025" containerID="7532e97c247706eb1ad257d9b5fad3471d511be1b98f386a3b2ee5bdc5906191" exitCode=0
Oct 31 19:48:56.804175 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:56.804049    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw" event={"ID":"1572ca38-17ef-41d9-b56a-a04bb167c025","Type":"ContainerDied","Data":"7532e97c247706eb1ad257d9b5fad3471d511be1b98f386a3b2ee5bdc5906191"}
Oct 31 19:48:56.889596 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:56.889571    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:56.999755 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:56.999730    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1572ca38-17ef-41d9-b56a-a04bb167c025-kube-api-access-jx5vb\" (UniqueName: \"kubernetes.io/projected/1572ca38-17ef-41d9-b56a-a04bb167c025-kube-api-access-jx5vb\") pod \"1572ca38-17ef-41d9-b56a-a04bb167c025\" (UID: \"1572ca38-17ef-41d9-b56a-a04bb167c025\") "
Oct 31 19:48:56.999901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:56.999779    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-webhook-cert\" (UniqueName: \"kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-webhook-cert\") pod \"1572ca38-17ef-41d9-b56a-a04bb167c025\" (UID: \"1572ca38-17ef-41d9-b56a-a04bb167c025\") "
Oct 31 19:48:56.999901 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:56.999809    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-metrics-certs\") pod \"1572ca38-17ef-41d9-b56a-a04bb167c025\" (UID: \"1572ca38-17ef-41d9-b56a-a04bb167c025\") "
Oct 31 19:48:57.002011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:57.001977    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-webhook-cert" pod "1572ca38-17ef-41d9-b56a-a04bb167c025" (UID: "1572ca38-17ef-41d9-b56a-a04bb167c025"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:57.002127 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:57.002016    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1572ca38-17ef-41d9-b56a-a04bb167c025-kube-api-access-jx5vb" pod "1572ca38-17ef-41d9-b56a-a04bb167c025" (UID: "1572ca38-17ef-41d9-b56a-a04bb167c025"). InnerVolumeSpecName "kube-api-access-jx5vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:57.002127 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:57.002059    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-metrics-certs" pod "1572ca38-17ef-41d9-b56a-a04bb167c025" (UID: "1572ca38-17ef-41d9-b56a-a04bb167c025"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:48:57.100984 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:57.100955    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx5vb\" (UniqueName: \"kubernetes.io/projected/1572ca38-17ef-41d9-b56a-a04bb167c025-kube-api-access-jx5vb\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:57.100984 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:57.100980    4484 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-webhook-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:57.100984 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:57.100990    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1572ca38-17ef-41d9-b56a-a04bb167c025-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:48:57.810791 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:57.810761    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw"
Oct 31 19:48:57.811257 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:57.810788    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fbfd7b7f-7zfzw" event={"ID":"1572ca38-17ef-41d9-b56a-a04bb167c025","Type":"ContainerDied","Data":"c710a5e2632e3156c2313c5980149d6a84f5990adc00762477543d6b9bdaeb3a"}
Oct 31 19:48:57.811257 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:57.810851    4484 scope.go:122] "RemoveContainer" containerID="7532e97c247706eb1ad257d9b5fad3471d511be1b98f386a3b2ee5bdc5906191"
Oct 31 19:48:57.978357 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:57.978323    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-7zfzw"]
Oct 31 19:48:58.028999 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:58.028964    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-7zfzw"]
Oct 31 19:48:59.433598 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.433564    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-5rqsl"]
Oct 31 19:48:59.434004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.433942    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a277764d-6c36-4e11-8b77-5d39ba95c0c1" containerName="revoke-keys"
Oct 31 19:48:59.434004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.433952    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a277764d-6c36-4e11-8b77-5d39ba95c0c1" containerName="revoke-keys"
Oct 31 19:48:59.434004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.433966    4484 cpu_manager.go:414] "containerMap: removing container" podUID="1572ca38-17ef-41d9-b56a-a04bb167c025" containerName="manager"
Oct 31 19:48:59.434004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.433971    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1572ca38-17ef-41d9-b56a-a04bb167c025" containerName="manager"
Oct 31 19:48:59.434142 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.434070    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a277764d-6c36-4e11-8b77-5d39ba95c0c1" containerName="revoke-keys"
Oct 31 19:48:59.434142 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.434078    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1572ca38-17ef-41d9-b56a-a04bb167c025" containerName="manager"
Oct 31 19:48:59.438342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.438325    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:48:59.543279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.543247    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-clhwj\""
Oct 31 19:48:59.543279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.543247    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 19:48:59.543589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.543319    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 19:48:59.543589 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.543323    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 19:48:59.545506 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.545465    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-5rqsl"]
Oct 31 19:48:59.624498 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.624452    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2bdb1a0-9e75-487d-a766-1c7bd313e7c4-webhook-cert\") pod \"maas-controller-64fbfd7b7f-5rqsl\" (UID: \"f2bdb1a0-9e75-487d-a766-1c7bd313e7c4\") " pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:48:59.624671 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.624565    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bdb1a0-9e75-487d-a766-1c7bd313e7c4-metrics-certs\") pod \"maas-controller-64fbfd7b7f-5rqsl\" (UID: \"f2bdb1a0-9e75-487d-a766-1c7bd313e7c4\") " pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:48:59.624671 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.624650    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8zwr\" (UniqueName: \"kubernetes.io/projected/f2bdb1a0-9e75-487d-a766-1c7bd313e7c4-kube-api-access-q8zwr\") pod \"maas-controller-64fbfd7b7f-5rqsl\" (UID: \"f2bdb1a0-9e75-487d-a766-1c7bd313e7c4\") " pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:48:59.725395 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.725305    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bdb1a0-9e75-487d-a766-1c7bd313e7c4-metrics-certs\") pod \"maas-controller-64fbfd7b7f-5rqsl\" (UID: \"f2bdb1a0-9e75-487d-a766-1c7bd313e7c4\") " pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:48:59.725395 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.725356    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8zwr\" (UniqueName: \"kubernetes.io/projected/f2bdb1a0-9e75-487d-a766-1c7bd313e7c4-kube-api-access-q8zwr\") pod \"maas-controller-64fbfd7b7f-5rqsl\" (UID: \"f2bdb1a0-9e75-487d-a766-1c7bd313e7c4\") " pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:48:59.725870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.725506    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2bdb1a0-9e75-487d-a766-1c7bd313e7c4-webhook-cert\") pod \"maas-controller-64fbfd7b7f-5rqsl\" (UID: \"f2bdb1a0-9e75-487d-a766-1c7bd313e7c4\") " pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:48:59.728117 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.728091    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2bdb1a0-9e75-487d-a766-1c7bd313e7c4-webhook-cert\") pod \"maas-controller-64fbfd7b7f-5rqsl\" (UID: \"f2bdb1a0-9e75-487d-a766-1c7bd313e7c4\") " pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:48:59.728178 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.728091    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bdb1a0-9e75-487d-a766-1c7bd313e7c4-metrics-certs\") pod \"maas-controller-64fbfd7b7f-5rqsl\" (UID: \"f2bdb1a0-9e75-487d-a766-1c7bd313e7c4\") " pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:48:59.784427 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.784401    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8zwr\" (UniqueName: \"kubernetes.io/projected/f2bdb1a0-9e75-487d-a766-1c7bd313e7c4-kube-api-access-q8zwr\") pod \"maas-controller-64fbfd7b7f-5rqsl\" (UID: \"f2bdb1a0-9e75-487d-a766-1c7bd313e7c4\") " pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:48:59.991984 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:48:59.991954    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1572ca38-17ef-41d9-b56a-a04bb167c025" path="/var/lib/kubelet/pods/1572ca38-17ef-41d9-b56a-a04bb167c025/volumes"
Oct 31 19:49:00.050457 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:00.050428    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:49:00.422080 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:00.422051    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64fbfd7b7f-5rqsl"]
Oct 31 19:49:00.423461 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:49:00.423431    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2bdb1a0_9e75_487d_a766_1c7bd313e7c4.slice/crio-d8a826708c4215a02682795ce53b98273c88252b2b3e1ad2b3ab6892c6ff362f WatchSource:0}: Error finding container d8a826708c4215a02682795ce53b98273c88252b2b3e1ad2b3ab6892c6ff362f: Status 404 returned error can't find the container with id d8a826708c4215a02682795ce53b98273c88252b2b3e1ad2b3ab6892c6ff362f
Oct 31 19:49:00.827340 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:00.827307    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl" event={"ID":"f2bdb1a0-9e75-487d-a766-1c7bd313e7c4","Type":"ContainerStarted","Data":"d8a826708c4215a02682795ce53b98273c88252b2b3e1ad2b3ab6892c6ff362f"}
Oct 31 19:49:01.837159 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:01.837124    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl" event={"ID":"f2bdb1a0-9e75-487d-a766-1c7bd313e7c4","Type":"ContainerStarted","Data":"1f12e6ceffc88ac3b4fbd51bebdc2bb079bd27e538557b0b2993d0c719abf232"}
Oct 31 19:49:01.837635 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:01.837218    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:49:01.956135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:01.956086    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl" podStartSLOduration=1.960491351 podStartE2EDuration="2.956072964s" totalImagesPullingTime="995.581613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:01.956052872 +0000 UTC m=+5288.102502737" watchObservedRunningTime="2026-10-31 19:49:01.956072964 +0000 UTC m=+5288.102522824"
Oct 31 19:49:08.046199 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:08.046163    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"]
Oct 31 19:49:08.097034 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:08.097001    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccf9fefc-fkg48"]
Oct 31 19:49:09.992114 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:09.992081    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca0e51a-2c91-4c8e-ac54-e682fe88271a" path="/var/lib/kubelet/pods/cca0e51a-2c91-4c8e-ac54-e682fe88271a/volumes"
Oct 31 19:49:12.844521 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:12.844487    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64fbfd7b7f-5rqsl"
Oct 31 19:49:25.378863 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:25.378825    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:49:25.467366 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:49:25.467336    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:49:30.929050 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:30.929014    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"]
Oct 31 19:49:30.932770 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:30.932754    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:30.989395 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:30.989361    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"]
Oct 31 19:49:31.001105 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.001082    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdmt\" (UniqueName: \"kubernetes.io/projected/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-kube-api-access-9rdmt\") pod \"maas-api-revoke-sub-f904bff9-pn286\" (UID: \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:31.001265 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.001112    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-pn286\" (UID: \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:31.001265 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.001141    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-pn286\" (UID: \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:31.037463 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.037435    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:49:31.037695 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.037677    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:49:31.102549 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.102522    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdmt\" (UniqueName: \"kubernetes.io/projected/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-kube-api-access-9rdmt\") pod \"maas-api-revoke-sub-f904bff9-pn286\" (UID: \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:31.102743 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.102557    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-pn286\" (UID: \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:31.102743 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.102584    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-pn286\" (UID: \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:31.103210 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.103189    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-pn286\" (UID: \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:31.105198 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.105167    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-pn286\" (UID: \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:31.163116 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.163086    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdmt\" (UniqueName: \"kubernetes.io/projected/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-kube-api-access-9rdmt\") pod \"maas-api-revoke-sub-f904bff9-pn286\" (UID: \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:31.245405 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.245377    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:31.613082 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.613056    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"]
Oct 31 19:49:31.614624 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:49:31.614596    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ee7d45_c55b_4db6_9c9f_a8b239350cd9.slice/crio-0f98b255192b502a926f81a4aa76fc3b3320e4f26eff567cfd872d28348cac8d WatchSource:0}: Error finding container 0f98b255192b502a926f81a4aa76fc3b3320e4f26eff567cfd872d28348cac8d: Status 404 returned error can't find the container with id 0f98b255192b502a926f81a4aa76fc3b3320e4f26eff567cfd872d28348cac8d
Oct 31 19:49:31.977096 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.977019    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286" event={"ID":"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9","Type":"ContainerStarted","Data":"b6872b5782e149c2fd3894e2cc0d4d7c97d8486c4b325d59ffc66fe8c0040f5b"}
Oct 31 19:49:31.977096 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:31.977069    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286" event={"ID":"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9","Type":"ContainerStarted","Data":"0f98b255192b502a926f81a4aa76fc3b3320e4f26eff567cfd872d28348cac8d"}
Oct 31 19:49:32.094817 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:32.094765    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286" podStartSLOduration=2.094751404 podStartE2EDuration="2.094751404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:32.09253384 +0000 UTC m=+5318.238983703" watchObservedRunningTime="2026-10-31 19:49:32.094751404 +0000 UTC m=+5318.241201267"
Oct 31 19:49:32.982694 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:32.982658    4484 generic.go:356] "Generic (PLEG): container finished" podID="e5ee7d45-c55b-4db6-9c9f-a8b239350cd9" containerID="b6872b5782e149c2fd3894e2cc0d4d7c97d8486c4b325d59ffc66fe8c0040f5b" exitCode=0
Oct 31 19:49:32.983082 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:32.982719    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286" event={"ID":"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9","Type":"ContainerDied","Data":"b6872b5782e149c2fd3894e2cc0d4d7c97d8486c4b325d59ffc66fe8c0040f5b"}
Oct 31 19:49:34.120840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.120814    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:34.234998 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.234968    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-xks-ca\") pod \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\" (UID: \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\") "
Oct 31 19:49:34.235152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.235032    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-openshift-ca\") pod \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\" (UID: \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\") "
Oct 31 19:49:34.235152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.235083    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-kube-api-access-9rdmt\" (UniqueName: \"kubernetes.io/projected/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-kube-api-access-9rdmt\") pod \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\" (UID: \"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9\") "
Oct 31 19:49:34.235342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.235322    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-openshift-ca" pod "e5ee7d45-c55b-4db6-9c9f-a8b239350cd9" (UID: "e5ee7d45-c55b-4db6-9c9f-a8b239350cd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:34.237773 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.237751    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-xks-ca" pod "e5ee7d45-c55b-4db6-9c9f-a8b239350cd9" (UID: "e5ee7d45-c55b-4db6-9c9f-a8b239350cd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:34.237914 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.237772    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-kube-api-access-9rdmt" pod "e5ee7d45-c55b-4db6-9c9f-a8b239350cd9" (UID: "e5ee7d45-c55b-4db6-9c9f-a8b239350cd9"). InnerVolumeSpecName "kube-api-access-9rdmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:34.336327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.336294    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:49:34.336327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.336325    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:49:34.336554 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.336340    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rdmt\" (UniqueName: \"kubernetes.io/projected/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9-kube-api-access-9rdmt\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:49:34.993916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.993879    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286" event={"ID":"e5ee7d45-c55b-4db6-9c9f-a8b239350cd9","Type":"ContainerDied","Data":"0f98b255192b502a926f81a4aa76fc3b3320e4f26eff567cfd872d28348cac8d"}
Oct 31 19:49:34.993916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.993907    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"
Oct 31 19:49:34.994132 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:34.993934    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f98b255192b502a926f81a4aa76fc3b3320e4f26eff567cfd872d28348cac8d"
Oct 31 19:49:43.861703 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:43.861666    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:49:44.045942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:44.045905    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"]
Oct 31 19:49:44.096537 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:44.096506    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14655279-p4n59"]
Oct 31 19:49:45.992682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:45.992643    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c4bb76-1f53-4b2f-92a3-b6e0bba1a137" path="/var/lib/kubelet/pods/40c4bb76-1f53-4b2f-92a3-b6e0bba1a137/volumes"
Oct 31 19:49:52.080315 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.080279    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"]
Oct 31 19:49:52.080783 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.080681    4484 cpu_manager.go:414] "containerMap: removing container" podUID="e5ee7d45-c55b-4db6-9c9f-a8b239350cd9" containerName="revoke-keys"
Oct 31 19:49:52.080783 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.080693    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ee7d45-c55b-4db6-9c9f-a8b239350cd9" containerName="revoke-keys"
Oct 31 19:49:52.080783 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.080780    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ee7d45-c55b-4db6-9c9f-a8b239350cd9" containerName="revoke-keys"
Oct 31 19:49:52.083952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.083935    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.193363 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.193327    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"]
Oct 31 19:49:52.202756 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.202726    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.202924 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.202792    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9f80747-2277-4637-a04a-15207891a808-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.202924 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.202884    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.203066 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.202942    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9f80747-2277-4637-a04a-15207891a808-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.203066 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.203035    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkv82\" (UniqueName: \"kubernetes.io/projected/a9f80747-2277-4637-a04a-15207891a808-kube-api-access-dkv82\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.203196 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.203091    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.203260 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.203184    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.203260 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.203243    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.203451 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.203274    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9f80747-2277-4637-a04a-15207891a808-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.303941 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.303901    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9f80747-2277-4637-a04a-15207891a808-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.303941 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.303954    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.304222 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.303990    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9f80747-2277-4637-a04a-15207891a808-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.304222 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.304014    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkv82\" (UniqueName: \"kubernetes.io/projected/a9f80747-2277-4637-a04a-15207891a808-kube-api-access-dkv82\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.304222 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.304051    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.304222 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.304085    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.304447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.304247    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.304447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.304375    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9f80747-2277-4637-a04a-15207891a808-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.304582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.304439    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.304582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.304531    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.304582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.304537    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.304884 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.304863    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.305003 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.304989    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.305369 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.305347    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9f80747-2277-4637-a04a-15207891a808-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.306735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.306711    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9f80747-2277-4637-a04a-15207891a808-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.306987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.306957    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9f80747-2277-4637-a04a-15207891a808-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.362378 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.362313    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9f80747-2277-4637-a04a-15207891a808-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.362827 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.362807    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkv82\" (UniqueName: \"kubernetes.io/projected/a9f80747-2277-4637-a04a-15207891a808-kube-api-access-dkv82\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vnvmf\" (UID: \"a9f80747-2277-4637-a04a-15207891a808\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.397310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.397274    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:52.824342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.824317    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"]
Oct 31 19:49:52.825805 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:49:52.825779    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f80747_2277_4637_a04a_15207891a808.slice/crio-a4f0b4f2933100330db7b396628f4b248cac7487a71bb75930e450fffa22e76c WatchSource:0}: Error finding container a4f0b4f2933100330db7b396628f4b248cac7487a71bb75930e450fffa22e76c: Status 404 returned error can't find the container with id a4f0b4f2933100330db7b396628f4b248cac7487a71bb75930e450fffa22e76c
Oct 31 19:49:52.827921 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.827890    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:49:52.827973 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.827957    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:49:52.828013 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:52.827987    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:49:53.085286 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:53.085200    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:49:53.103812 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:53.103781    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" event={"ID":"a9f80747-2277-4637-a04a-15207891a808","Type":"ContainerStarted","Data":"23528ab9beb2f030f23d7b4988f9f26e3f49bedc00d4b5044e641ee143a9b31e"}
Oct 31 19:49:53.104008 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:53.103831    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" event={"ID":"a9f80747-2277-4637-a04a-15207891a808","Type":"ContainerStarted","Data":"a4f0b4f2933100330db7b396628f4b248cac7487a71bb75930e450fffa22e76c"}
Oct 31 19:49:53.224855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:53.224801    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podStartSLOduration=1.224788107 podStartE2EDuration="1.224788107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:53.224358419 +0000 UTC m=+5339.370808283" watchObservedRunningTime="2026-10-31 19:49:53.224788107 +0000 UTC m=+5339.371237971"
Oct 31 19:49:53.398031 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:53.397934    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:49:53.399787 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:53.399765    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vnvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 19:49:53.399849 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:53.399818    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podUID="a9f80747-2277-4637-a04a-15207891a808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 19:49:54.397999 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:54.397962    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vnvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 19:49:54.398427 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:54.398014    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podUID="a9f80747-2277-4637-a04a-15207891a808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 19:49:55.046178 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:55.046140    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"]
Oct 31 19:49:55.097174 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:55.097129    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d270138-bg6c9"]
Oct 31 19:49:55.398538 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:55.398435    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vnvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 19:49:55.398538 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:55.398507    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podUID="a9f80747-2277-4637-a04a-15207891a808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 19:49:55.490516 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:49:55.490450    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:49:55.994885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:55.994850    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd" path="/var/lib/kubelet/pods/59d5f86a-5f66-4c7e-8e8e-e134ec6eefbd/volumes"
Oct 31 19:49:56.087696 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:56.087660    4484 scope.go:122] "RemoveContainer" containerID="9f17774abc28d4cae886197d517cd47322e79ee1f61405fdbc9f84313637ae77"
Oct 31 19:49:56.398069 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:56.397983    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vnvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 19:49:56.398069 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:56.398038    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podUID="a9f80747-2277-4637-a04a-15207891a808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 19:49:57.398569 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:57.398533    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vnvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 19:49:57.398973 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:57.398591    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podUID="a9f80747-2277-4637-a04a-15207891a808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 19:49:58.397938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:58.397900    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vnvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 19:49:58.398103 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:58.397963    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podUID="a9f80747-2277-4637-a04a-15207891a808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 19:49:59.398319 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:59.398278    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vnvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body=
Oct 31 19:49:59.398888 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:49:59.398346    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podUID="a9f80747-2277-4637-a04a-15207891a808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused"
Oct 31 19:50:01.398389 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:01.398351    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vnvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:50:01.398876 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:01.398427    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podUID="a9f80747-2277-4637-a04a-15207891a808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:02.439759 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:02.439714    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vnvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:50:02.440322 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:02.439781    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podUID="a9f80747-2277-4637-a04a-15207891a808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:02.693163 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:02.693057    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:50:02.899644 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:02.899607    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:50:02.899952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:02.899928    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:50:02.900661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:02.900640    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf"
Oct 31 19:50:04.427155 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.427113    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"]
Oct 31 19:50:04.432165 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.432142    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:04.486160 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.486135    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:50:04.514012 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.513981    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mqwmj\" (UID: \"1805861e-833d-415d-82f1-bdb8aac7d5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:04.514012 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.514013    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mqwmj\" (UID: \"1805861e-833d-415d-82f1-bdb8aac7d5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:04.514219 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.514040    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5pnn\" (UniqueName: \"kubernetes.io/projected/1805861e-833d-415d-82f1-bdb8aac7d5c3-kube-api-access-q5pnn\") pod \"maas-api-revoke-sub-f03629db-mqwmj\" (UID: \"1805861e-833d-415d-82f1-bdb8aac7d5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:04.537154 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.537129    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:50:04.538763 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.538744    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"]
Oct 31 19:50:04.614784 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.614750    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mqwmj\" (UID: \"1805861e-833d-415d-82f1-bdb8aac7d5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:04.614964 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.614789    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mqwmj\" (UID: \"1805861e-833d-415d-82f1-bdb8aac7d5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:04.614964 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.614836    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5pnn\" (UniqueName: \"kubernetes.io/projected/1805861e-833d-415d-82f1-bdb8aac7d5c3-kube-api-access-q5pnn\") pod \"maas-api-revoke-sub-f03629db-mqwmj\" (UID: \"1805861e-833d-415d-82f1-bdb8aac7d5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:04.615453 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.615420    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mqwmj\" (UID: \"1805861e-833d-415d-82f1-bdb8aac7d5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:04.617229 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.617211    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mqwmj\" (UID: \"1805861e-833d-415d-82f1-bdb8aac7d5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:04.673601 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.673575    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5pnn\" (UniqueName: \"kubernetes.io/projected/1805861e-833d-415d-82f1-bdb8aac7d5c3-kube-api-access-q5pnn\") pod \"maas-api-revoke-sub-f03629db-mqwmj\" (UID: \"1805861e-833d-415d-82f1-bdb8aac7d5c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:04.744233 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:04.744206    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:05.118769 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:05.118739    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"]
Oct 31 19:50:05.120399 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:50:05.120372    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1805861e_833d_415d_82f1_bdb8aac7d5c3.slice/crio-9a233ef58962378d26ec2e8cbc3aef661b3516b4bdf901ca1bd034ebde54df06 WatchSource:0}: Error finding container 9a233ef58962378d26ec2e8cbc3aef661b3516b4bdf901ca1bd034ebde54df06: Status 404 returned error can't find the container with id 9a233ef58962378d26ec2e8cbc3aef661b3516b4bdf901ca1bd034ebde54df06
Oct 31 19:50:05.171388 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:05.171359    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj" event={"ID":"1805861e-833d-415d-82f1-bdb8aac7d5c3","Type":"ContainerStarted","Data":"9a233ef58962378d26ec2e8cbc3aef661b3516b4bdf901ca1bd034ebde54df06"}
Oct 31 19:50:06.177411 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:06.177319    4484 generic.go:356] "Generic (PLEG): container finished" podID="1805861e-833d-415d-82f1-bdb8aac7d5c3" containerID="08ff3502933a284dfb70f1b49c976ffa1576d0575d8e6e231ce6eab9c262b339" exitCode=0
Oct 31 19:50:06.177905 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:06.177412    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj" event={"ID":"1805861e-833d-415d-82f1-bdb8aac7d5c3","Type":"ContainerDied","Data":"08ff3502933a284dfb70f1b49c976ffa1576d0575d8e6e231ce6eab9c262b339"}
Oct 31 19:50:07.315291 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:07.315250    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:07.340303 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:07.340277    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-xks-ca\") pod \"1805861e-833d-415d-82f1-bdb8aac7d5c3\" (UID: \"1805861e-833d-415d-82f1-bdb8aac7d5c3\") "
Oct 31 19:50:07.340398 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:07.340309    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1805861e-833d-415d-82f1-bdb8aac7d5c3-kube-api-access-q5pnn\" (UniqueName: \"kubernetes.io/projected/1805861e-833d-415d-82f1-bdb8aac7d5c3-kube-api-access-q5pnn\") pod \"1805861e-833d-415d-82f1-bdb8aac7d5c3\" (UID: \"1805861e-833d-415d-82f1-bdb8aac7d5c3\") "
Oct 31 19:50:07.340398 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:07.340351    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-openshift-ca\") pod \"1805861e-833d-415d-82f1-bdb8aac7d5c3\" (UID: \"1805861e-833d-415d-82f1-bdb8aac7d5c3\") "
Oct 31 19:50:07.340742 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:07.340717    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-openshift-ca" pod "1805861e-833d-415d-82f1-bdb8aac7d5c3" (UID: "1805861e-833d-415d-82f1-bdb8aac7d5c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:07.342523 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:07.342494    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-xks-ca" pod "1805861e-833d-415d-82f1-bdb8aac7d5c3" (UID: "1805861e-833d-415d-82f1-bdb8aac7d5c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:07.342614 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:07.342552    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1805861e-833d-415d-82f1-bdb8aac7d5c3-kube-api-access-q5pnn" pod "1805861e-833d-415d-82f1-bdb8aac7d5c3" (UID: "1805861e-833d-415d-82f1-bdb8aac7d5c3"). InnerVolumeSpecName "kube-api-access-q5pnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:07.441942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:07.441875    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:50:07.441942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:07.441901    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5pnn\" (UniqueName: \"kubernetes.io/projected/1805861e-833d-415d-82f1-bdb8aac7d5c3-kube-api-access-q5pnn\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:50:07.441942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:07.441911    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1805861e-833d-415d-82f1-bdb8aac7d5c3-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:50:08.188246 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:08.188219    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"
Oct 31 19:50:08.188433 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:08.188244    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj" event={"ID":"1805861e-833d-415d-82f1-bdb8aac7d5c3","Type":"ContainerDied","Data":"9a233ef58962378d26ec2e8cbc3aef661b3516b4bdf901ca1bd034ebde54df06"}
Oct 31 19:50:08.188433 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:08.188300    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a233ef58962378d26ec2e8cbc3aef661b3516b4bdf901ca1bd034ebde54df06"
Oct 31 19:50:11.256883 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:11.256851    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:50:22.342057 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.342021    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"]
Oct 31 19:50:22.342460 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.342416    4484 cpu_manager.go:414] "containerMap: removing container" podUID="1805861e-833d-415d-82f1-bdb8aac7d5c3" containerName="revoke-keys"
Oct 31 19:50:22.342460 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.342426    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1805861e-833d-415d-82f1-bdb8aac7d5c3" containerName="revoke-keys"
Oct 31 19:50:22.342641 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.342510    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1805861e-833d-415d-82f1-bdb8aac7d5c3" containerName="revoke-keys"
Oct 31 19:50:22.345682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.345664    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.522499 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.497125    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"]
Oct 31 19:50:22.523001 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.522969    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.523733 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.523699    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcf339-7a9f-47df-acef-a2ab94742e15-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.524000 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.523972    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcf339-7a9f-47df-acef-a2ab94742e15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.524270 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.524243    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4gk\" (UniqueName: \"kubernetes.io/projected/a7dcf339-7a9f-47df-acef-a2ab94742e15-kube-api-access-zw4gk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.524665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.524648    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.524980 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.524962    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.525142 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.525125    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcf339-7a9f-47df-acef-a2ab94742e15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.525427 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.525410    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.525605 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.525587    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.627170 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.627027    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.628785 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.628121    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.629197 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.629172    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcf339-7a9f-47df-acef-a2ab94742e15-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.629262 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.629247    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcf339-7a9f-47df-acef-a2ab94742e15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.629325 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.629309    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4gk\" (UniqueName: \"kubernetes.io/projected/a7dcf339-7a9f-47df-acef-a2ab94742e15-kube-api-access-zw4gk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.629408 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.629394    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.629447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.629428    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.629511 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.629453    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcf339-7a9f-47df-acef-a2ab94742e15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.629547 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.629506    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.629547 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.629538    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.630672 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.630642    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7dcf339-7a9f-47df-acef-a2ab94742e15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.630857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.630730    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.631115 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.631097    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.631669 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.631570    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.633171 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.633143    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7dcf339-7a9f-47df-acef-a2ab94742e15-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.634058 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.634039    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7dcf339-7a9f-47df-acef-a2ab94742e15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.705640 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.705561    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7dcf339-7a9f-47df-acef-a2ab94742e15-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.705822 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.705765    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4gk\" (UniqueName: \"kubernetes.io/projected/a7dcf339-7a9f-47df-acef-a2ab94742e15-kube-api-access-zw4gk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-g4xdk\" (UID: \"a7dcf339-7a9f-47df-acef-a2ab94742e15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:22.958550 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:22.958409    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:23.419096 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:23.419072    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"]
Oct 31 19:50:23.421045 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:50:23.421008    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7dcf339_7a9f_47df_acef_a2ab94742e15.slice/crio-988436d339d760661162e0948bd4383c8b2006e343d7ef76137e604ea6c01137 WatchSource:0}: Error finding container 988436d339d760661162e0948bd4383c8b2006e343d7ef76137e604ea6c01137: Status 404 returned error can't find the container with id 988436d339d760661162e0948bd4383c8b2006e343d7ef76137e604ea6c01137
Oct 31 19:50:23.423183 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:23.423151    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:50:23.423271 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:23.423215    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:50:23.423271 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:23.423243    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 19:50:24.132592 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.132557    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"]
Oct 31 19:50:24.136199 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.136179    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:24.147360 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.147332    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdgqg\" (UniqueName: \"kubernetes.io/projected/9d80a47f-8692-434d-800d-b9ebc97dea05-kube-api-access-tdgqg\") pod \"maas-api-revoke-sub-0165d7d2-s7lrm\" (UID: \"9d80a47f-8692-434d-800d-b9ebc97dea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:24.147501 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.147459    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-s7lrm\" (UID: \"9d80a47f-8692-434d-800d-b9ebc97dea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:24.147582 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.147568    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-s7lrm\" (UID: \"9d80a47f-8692-434d-800d-b9ebc97dea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:24.241750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.241720    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:50:24.241947 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.241724    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:50:24.243781 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.243759    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"]
Oct 31 19:50:24.248708 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.248686    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-s7lrm\" (UID: \"9d80a47f-8692-434d-800d-b9ebc97dea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:24.248786 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.248738    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdgqg\" (UniqueName: \"kubernetes.io/projected/9d80a47f-8692-434d-800d-b9ebc97dea05-kube-api-access-tdgqg\") pod \"maas-api-revoke-sub-0165d7d2-s7lrm\" (UID: \"9d80a47f-8692-434d-800d-b9ebc97dea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:24.248857 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.248800    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-s7lrm\" (UID: \"9d80a47f-8692-434d-800d-b9ebc97dea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:24.249395 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.249376    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-s7lrm\" (UID: \"9d80a47f-8692-434d-800d-b9ebc97dea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:24.251097 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.251079    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-s7lrm\" (UID: \"9d80a47f-8692-434d-800d-b9ebc97dea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:24.290500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.290436    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" event={"ID":"a7dcf339-7a9f-47df-acef-a2ab94742e15","Type":"ContainerStarted","Data":"d9fe4b5af6b4c2a75825e1f39e266ea5f3336c30001cd049785f2d8a2d9967f8"}
Oct 31 19:50:24.290693 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.290535    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" event={"ID":"a7dcf339-7a9f-47df-acef-a2ab94742e15","Type":"ContainerStarted","Data":"988436d339d760661162e0948bd4383c8b2006e343d7ef76137e604ea6c01137"}
Oct 31 19:50:24.308187 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.308163    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdgqg\" (UniqueName: \"kubernetes.io/projected/9d80a47f-8692-434d-800d-b9ebc97dea05-kube-api-access-tdgqg\") pod \"maas-api-revoke-sub-0165d7d2-s7lrm\" (UID: \"9d80a47f-8692-434d-800d-b9ebc97dea05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:24.411594 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.411467    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" podStartSLOduration=2.41145368 podStartE2EDuration="2.41145368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:24.410463101 +0000 UTC m=+5370.556912966" watchObservedRunningTime="2026-10-31 19:50:24.41145368 +0000 UTC m=+5370.557903609"
Oct 31 19:50:24.449522 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.449458    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:24.844254 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.844229    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"]
Oct 31 19:50:24.845872 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:50:24.845843    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d80a47f_8692_434d_800d_b9ebc97dea05.slice/crio-dbbcdaa4c7704b1c05a2734f7708ba357a94dce2c7752b795ccf9dc543419ca4 WatchSource:0}: Error finding container dbbcdaa4c7704b1c05a2734f7708ba357a94dce2c7752b795ccf9dc543419ca4: Status 404 returned error can't find the container with id dbbcdaa4c7704b1c05a2734f7708ba357a94dce2c7752b795ccf9dc543419ca4
Oct 31 19:50:24.959441 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.959413    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:24.964518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.961048    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-g4xdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 19:50:24.964518 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:24.961101    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" podUID="a7dcf339-7a9f-47df-acef-a2ab94742e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 19:50:25.298155 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:25.298108    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm" event={"ID":"9d80a47f-8692-434d-800d-b9ebc97dea05","Type":"ContainerStarted","Data":"958deb839eebee9798d788981dc63ebc58046b1e6064a2e5540f112ddef30488"}
Oct 31 19:50:25.298336 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:25.298225    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm" event={"ID":"9d80a47f-8692-434d-800d-b9ebc97dea05","Type":"ContainerStarted","Data":"dbbcdaa4c7704b1c05a2734f7708ba357a94dce2c7752b795ccf9dc543419ca4"}
Oct 31 19:50:25.416245 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:25.416136    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm" podStartSLOduration=1.416114801 podStartE2EDuration="1.416114801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:25.414895399 +0000 UTC m=+5371.561345263" watchObservedRunningTime="2026-10-31 19:50:25.416114801 +0000 UTC m=+5371.562564667"
Oct 31 19:50:25.494892 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:50:25.494854    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:50:25.959267 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:25.959235    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-g4xdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 19:50:25.959442 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:25.959292    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" podUID="a7dcf339-7a9f-47df-acef-a2ab94742e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 19:50:26.305263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:26.305232    4484 generic.go:356] "Generic (PLEG): container finished" podID="9d80a47f-8692-434d-800d-b9ebc97dea05" containerID="958deb839eebee9798d788981dc63ebc58046b1e6064a2e5540f112ddef30488" exitCode=0
Oct 31 19:50:26.305460 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:26.305318    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm" event={"ID":"9d80a47f-8692-434d-800d-b9ebc97dea05","Type":"ContainerDied","Data":"958deb839eebee9798d788981dc63ebc58046b1e6064a2e5540f112ddef30488"}
Oct 31 19:50:26.959021 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:26.958983    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-g4xdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 19:50:26.959544 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:26.959041    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" podUID="a7dcf339-7a9f-47df-acef-a2ab94742e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 19:50:27.494855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:27.494831    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:27.601273 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:27.601183    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-xks-ca\") pod \"9d80a47f-8692-434d-800d-b9ebc97dea05\" (UID: \"9d80a47f-8692-434d-800d-b9ebc97dea05\") "
Oct 31 19:50:27.601273 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:27.601228    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-openshift-ca\") pod \"9d80a47f-8692-434d-800d-b9ebc97dea05\" (UID: \"9d80a47f-8692-434d-800d-b9ebc97dea05\") "
Oct 31 19:50:27.601273 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:27.601263    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d80a47f-8692-434d-800d-b9ebc97dea05-kube-api-access-tdgqg\" (UniqueName: \"kubernetes.io/projected/9d80a47f-8692-434d-800d-b9ebc97dea05-kube-api-access-tdgqg\") pod \"9d80a47f-8692-434d-800d-b9ebc97dea05\" (UID: \"9d80a47f-8692-434d-800d-b9ebc97dea05\") "
Oct 31 19:50:27.601670 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:27.601634    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-openshift-ca" pod "9d80a47f-8692-434d-800d-b9ebc97dea05" (UID: "9d80a47f-8692-434d-800d-b9ebc97dea05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:27.603585 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:27.603558    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d80a47f-8692-434d-800d-b9ebc97dea05-kube-api-access-tdgqg" pod "9d80a47f-8692-434d-800d-b9ebc97dea05" (UID: "9d80a47f-8692-434d-800d-b9ebc97dea05"). InnerVolumeSpecName "kube-api-access-tdgqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:27.603678 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:27.603582    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-xks-ca" pod "9d80a47f-8692-434d-800d-b9ebc97dea05" (UID: "9d80a47f-8692-434d-800d-b9ebc97dea05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:27.702383 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:27.702346    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:50:27.702383 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:27.702377    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d80a47f-8692-434d-800d-b9ebc97dea05-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:50:27.702383 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:27.702393    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdgqg\" (UniqueName: \"kubernetes.io/projected/9d80a47f-8692-434d-800d-b9ebc97dea05-kube-api-access-tdgqg\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:50:27.959448 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:27.959354    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-g4xdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 19:50:27.959448 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:27.959425    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" podUID="a7dcf339-7a9f-47df-acef-a2ab94742e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 19:50:28.315886 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:28.315857    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"
Oct 31 19:50:28.316067 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:28.315857    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm" event={"ID":"9d80a47f-8692-434d-800d-b9ebc97dea05","Type":"ContainerDied","Data":"dbbcdaa4c7704b1c05a2734f7708ba357a94dce2c7752b795ccf9dc543419ca4"}
Oct 31 19:50:28.316067 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:28.315990    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbbcdaa4c7704b1c05a2734f7708ba357a94dce2c7752b795ccf9dc543419ca4"
Oct 31 19:50:28.958817 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:28.958781    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-g4xdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 19:50:28.958996 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:28.958852    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" podUID="a7dcf339-7a9f-47df-acef-a2ab94742e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 19:50:29.959223 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:29.959188    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-g4xdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 31 19:50:29.959643 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:29.959273    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" podUID="a7dcf339-7a9f-47df-acef-a2ab94742e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 31 19:50:31.408504 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:31.408440    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:31.408993 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:31.408853    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:31.409591 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:31.409566    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk"
Oct 31 19:50:36.021114 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.021080    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"]
Oct 31 19:50:36.021642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.021572    4484 cpu_manager.go:414] "containerMap: removing container" podUID="9d80a47f-8692-434d-800d-b9ebc97dea05" containerName="revoke-keys"
Oct 31 19:50:36.021642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.021586    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d80a47f-8692-434d-800d-b9ebc97dea05" containerName="revoke-keys"
Oct 31 19:50:36.021724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.021654    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d80a47f-8692-434d-800d-b9ebc97dea05" containerName="revoke-keys"
Oct 31 19:50:36.025176 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.025160    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:36.080672 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.080643    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:50:36.083768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.083738    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"]
Oct 31 19:50:36.088135 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.088110    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xh8r5\" (UID: \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:36.088260 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.088144    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844kk\" (UniqueName: \"kubernetes.io/projected/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-kube-api-access-844kk\") pod \"maas-api-revoke-sub-6b19338f-xh8r5\" (UID: \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:36.088260 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.088165    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xh8r5\" (UID: \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:36.131667 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.131637    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:50:36.188687 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.188657    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xh8r5\" (UID: \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:36.188885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.188696    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844kk\" (UniqueName: \"kubernetes.io/projected/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-kube-api-access-844kk\") pod \"maas-api-revoke-sub-6b19338f-xh8r5\" (UID: \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:36.188885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.188718    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xh8r5\" (UID: \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:36.189297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.189269    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xh8r5\" (UID: \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:36.191142 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.191117    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xh8r5\" (UID: \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:36.248680 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.248648    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-844kk\" (UniqueName: \"kubernetes.io/projected/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-kube-api-access-844kk\") pod \"maas-api-revoke-sub-6b19338f-xh8r5\" (UID: \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:36.338030 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.337947    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:36.714731 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:36.714706    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"]
Oct 31 19:50:36.716593 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:50:36.716564    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2872a3_4f1d_4c5b_9937_8ccc8777eebe.slice/crio-131b36a0079094d80179b6a67deddb7c8e4e7f7f73742c9ebee76fdda070e366 WatchSource:0}: Error finding container 131b36a0079094d80179b6a67deddb7c8e4e7f7f73742c9ebee76fdda070e366: Status 404 returned error can't find the container with id 131b36a0079094d80179b6a67deddb7c8e4e7f7f73742c9ebee76fdda070e366
Oct 31 19:50:37.364860 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:37.364829    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5" event={"ID":"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe","Type":"ContainerStarted","Data":"6d861385b3cbf620d8124088178b904cea71033d3357c09215f2088b88d2870d"}
Oct 31 19:50:37.365313 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:37.364890    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5" event={"ID":"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe","Type":"ContainerStarted","Data":"131b36a0079094d80179b6a67deddb7c8e4e7f7f73742c9ebee76fdda070e366"}
Oct 31 19:50:37.485151 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:37.485097    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5" podStartSLOduration=2.485082112 podStartE2EDuration="2.485082112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:37.48324854 +0000 UTC m=+5383.629698405" watchObservedRunningTime="2026-10-31 19:50:37.485082112 +0000 UTC m=+5383.631531998"
Oct 31 19:50:38.374078 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:38.374038    4484 generic.go:356] "Generic (PLEG): container finished" podID="9c2872a3-4f1d-4c5b-9937-8ccc8777eebe" containerID="6d861385b3cbf620d8124088178b904cea71033d3357c09215f2088b88d2870d" exitCode=0
Oct 31 19:50:38.374616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:38.374089    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5" event={"ID":"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe","Type":"ContainerDied","Data":"6d861385b3cbf620d8124088178b904cea71033d3357c09215f2088b88d2870d"}
Oct 31 19:50:39.519037 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:39.519011    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:39.628354 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:39.628326    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-kube-api-access-844kk\" (UniqueName: \"kubernetes.io/projected/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-kube-api-access-844kk\") pod \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\" (UID: \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\") "
Oct 31 19:50:39.628554 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:39.628400    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-xks-ca\") pod \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\" (UID: \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\") "
Oct 31 19:50:39.628554 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:39.628437    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-openshift-ca\") pod \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\" (UID: \"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe\") "
Oct 31 19:50:39.628865 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:39.628833    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-openshift-ca" pod "9c2872a3-4f1d-4c5b-9937-8ccc8777eebe" (UID: "9c2872a3-4f1d-4c5b-9937-8ccc8777eebe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:39.630674 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:39.630623    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-kube-api-access-844kk" pod "9c2872a3-4f1d-4c5b-9937-8ccc8777eebe" (UID: "9c2872a3-4f1d-4c5b-9937-8ccc8777eebe"). InnerVolumeSpecName "kube-api-access-844kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:39.630810 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:39.630788    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-xks-ca" pod "9c2872a3-4f1d-4c5b-9937-8ccc8777eebe" (UID: "9c2872a3-4f1d-4c5b-9937-8ccc8777eebe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:39.729123 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:39.729091    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-844kk\" (UniqueName: \"kubernetes.io/projected/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-kube-api-access-844kk\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:50:39.729123 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:39.729118    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:50:39.729123 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:39.729129    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:50:40.386017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:40.385942    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"
Oct 31 19:50:40.386017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:40.385969    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5" event={"ID":"9c2872a3-4f1d-4c5b-9937-8ccc8777eebe","Type":"ContainerDied","Data":"131b36a0079094d80179b6a67deddb7c8e4e7f7f73742c9ebee76fdda070e366"}
Oct 31 19:50:40.386017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:40.386013    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="131b36a0079094d80179b6a67deddb7c8e4e7f7f73742c9ebee76fdda070e366"
Oct 31 19:50:48.892331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:48.892293    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:50:52.164840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.164803    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"]
Oct 31 19:50:52.165246 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.165229    4484 cpu_manager.go:414] "containerMap: removing container" podUID="9c2872a3-4f1d-4c5b-9937-8ccc8777eebe" containerName="revoke-keys"
Oct 31 19:50:52.165246 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.165240    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2872a3-4f1d-4c5b-9937-8ccc8777eebe" containerName="revoke-keys"
Oct 31 19:50:52.165331 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.165317    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2872a3-4f1d-4c5b-9937-8ccc8777eebe" containerName="revoke-keys"
Oct 31 19:50:52.168448 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.168429    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:50:52.225620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.225585    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"]
Oct 31 19:50:52.252166 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.252131    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trjqf\" (UniqueName: \"kubernetes.io/projected/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-kube-api-access-trjqf\") pod \"e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm\" (UID: \"28d347a1-e182-4a41-9d1f-0ac9ffe842d8\") " pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:50:52.252346 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.252295    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-tls-certs\") pod \"e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm\" (UID: \"28d347a1-e182-4a41-9d1f-0ac9ffe842d8\") " pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:50:52.272922 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.272897    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-466e73b1-kserve-self-signed-certs\""
Oct 31 19:50:52.273027 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.272972    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-466e73b1-epp-sa-dockercfg-bsxh6\""
Oct 31 19:50:52.353027 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.352936    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-tls-certs\") pod \"e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm\" (UID: \"28d347a1-e182-4a41-9d1f-0ac9ffe842d8\") " pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:50:52.353227 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.353089    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trjqf\" (UniqueName: \"kubernetes.io/projected/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-kube-api-access-trjqf\") pod \"e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm\" (UID: \"28d347a1-e182-4a41-9d1f-0ac9ffe842d8\") " pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:50:52.356120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.356090    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-tls-certs\") pod \"e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm\" (UID: \"28d347a1-e182-4a41-9d1f-0ac9ffe842d8\") " pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:50:52.411305 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.411273    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trjqf\" (UniqueName: \"kubernetes.io/projected/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-kube-api-access-trjqf\") pod \"e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm\" (UID: \"28d347a1-e182-4a41-9d1f-0ac9ffe842d8\") " pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:50:52.481768 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.481694    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:50:52.857616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:52.857586    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"]
Oct 31 19:50:52.859100 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:50:52.859078    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d347a1_e182_4a41_9d1f_0ac9ffe842d8.slice/crio-b650fa08b3a43905b42d17ff7e89862eae1496171e428bb317ee46a8026d79f7 WatchSource:0}: Error finding container b650fa08b3a43905b42d17ff7e89862eae1496171e428bb317ee46a8026d79f7: Status 404 returned error can't find the container with id b650fa08b3a43905b42d17ff7e89862eae1496171e428bb317ee46a8026d79f7
Oct 31 19:50:53.453914 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:53.453878    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm" event={"ID":"28d347a1-e182-4a41-9d1f-0ac9ffe842d8","Type":"ContainerStarted","Data":"b650fa08b3a43905b42d17ff7e89862eae1496171e428bb317ee46a8026d79f7"}
Oct 31 19:50:54.984568 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:54.984543    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:50:54.989635 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:54.989615    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:50:55.467699 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:55.467660    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm" event={"ID":"28d347a1-e182-4a41-9d1f-0ac9ffe842d8","Type":"ContainerStarted","Data":"455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983"}
Oct 31 19:50:55.467889 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:55.467744    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:50:55.468387 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:50:55.468361    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:50:55.585713 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:55.585618    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm" podStartSLOduration=1.265013251 podStartE2EDuration="3.585604344s" totalImagesPullingTime="2.320591093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:55.5842415 +0000 UTC m=+5401.730691365" watchObservedRunningTime="2026-10-31 19:50:55.585604344 +0000 UTC m=+5401.732054207"
Oct 31 19:50:56.112146 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:56.112112    4484 scope.go:122] "RemoveContainer" containerID="c6d56b3b9f3d5d7cc18b5e966d946bfbd78c82940cebca1eb3b9e748d7053c4f"
Oct 31 19:50:56.123682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:56.123650    4484 scope.go:122] "RemoveContainer" containerID="27a311d1884fb600738b1ff94fee7fe256f360dfb0360eccb09e2c7b5490fccb"
Oct 31 19:50:56.132881 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:50:56.132855    4484 scope.go:122] "RemoveContainer" containerID="7532f6275b11968bd88dd8302b1e086287b35b07bad04b46f6337af9f7720861"
Oct 31 19:51:02.992483 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:51:02.992436    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:51:25.468127 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:51:25.468087    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:51:26.475654 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:51:26.475626    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:51:34.769201 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:51:34.769165    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:51:55.466652 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:51:55.466609    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:51:59.207021 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:51:59.206987    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"]
Oct 31 19:51:59.255907 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:51:59.255873    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:52:06.488500 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:06.486555    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm" podUID="28d347a1-e182-4a41-9d1f-0ac9ffe842d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:52:08.991553 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:08.991523    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:52:14.227907 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:14.227864    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm" podUID="28d347a1-e182-4a41-9d1f-0ac9ffe842d8" containerName="main" containerID="cri-o://455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983" gracePeriod=45
Oct 31 19:52:16.476366 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:16.476331    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm" podUID="28d347a1-e182-4a41-9d1f-0ac9ffe842d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:52:25.467076 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:52:25.467040    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:52:26.476092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:26.476054    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm" podUID="28d347a1-e182-4a41-9d1f-0ac9ffe842d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:52:26.476604 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:26.476157    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:52:36.476322 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:36.476285    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm" podUID="28d347a1-e182-4a41-9d1f-0ac9ffe842d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:52:37.672715 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:37.672673    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:52:44.469840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:44.469812    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:52:44.597367 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:44.597283    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-kube-api-access-trjqf\" (UniqueName: \"kubernetes.io/projected/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-kube-api-access-trjqf\") pod \"28d347a1-e182-4a41-9d1f-0ac9ffe842d8\" (UID: \"28d347a1-e182-4a41-9d1f-0ac9ffe842d8\") "
Oct 31 19:52:44.597367 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:44.597338    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-tls-certs\") pod \"28d347a1-e182-4a41-9d1f-0ac9ffe842d8\" (UID: \"28d347a1-e182-4a41-9d1f-0ac9ffe842d8\") "
Oct 31 19:52:44.599726 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:44.599695    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-tls-certs" pod "28d347a1-e182-4a41-9d1f-0ac9ffe842d8" (UID: "28d347a1-e182-4a41-9d1f-0ac9ffe842d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:52:44.599855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:44.599764    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-kube-api-access-trjqf" pod "28d347a1-e182-4a41-9d1f-0ac9ffe842d8" (UID: "28d347a1-e182-4a41-9d1f-0ac9ffe842d8"). InnerVolumeSpecName "kube-api-access-trjqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:52:44.698116 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:44.698079    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trjqf\" (UniqueName: \"kubernetes.io/projected/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-kube-api-access-trjqf\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:52:44.698116 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:44.698114    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28d347a1-e182-4a41-9d1f-0ac9ffe842d8-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:52:45.114766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:45.114736    4484 generic.go:356] "Generic (PLEG): container finished" podID="28d347a1-e182-4a41-9d1f-0ac9ffe842d8" containerID="455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983" exitCode=0
Oct 31 19:52:45.114945 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:45.114778    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"
Oct 31 19:52:45.114945 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:45.114815    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm" event={"ID":"28d347a1-e182-4a41-9d1f-0ac9ffe842d8","Type":"ContainerDied","Data":"455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983"}
Oct 31 19:52:45.114945 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:45.114875    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm" event={"ID":"28d347a1-e182-4a41-9d1f-0ac9ffe842d8","Type":"ContainerDied","Data":"b650fa08b3a43905b42d17ff7e89862eae1496171e428bb317ee46a8026d79f7"}
Oct 31 19:52:45.114945 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:45.114904    4484 scope.go:122] "RemoveContainer" containerID="455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983"
Oct 31 19:52:45.124232 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:45.124218    4484 scope.go:122] "RemoveContainer" containerID="455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983"
Oct 31 19:52:45.124529 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:52:45.124501    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983\": container with ID starting with 455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983 not found: ID does not exist" containerID="455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983"
Oct 31 19:52:45.124611 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:45.124540    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983"} err="failed to get container status \"455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983\": rpc error: code = NotFound desc = could not find container \"455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983\": container with ID starting with 455505afe86f1288604ad51c7d9e2b056314667728a0a864fa08ed865ea75983 not found: ID does not exist"
Oct 31 19:52:45.292949 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:45.292903    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"]
Oct 31 19:52:45.343180 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:45.343150    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-466e73b1-kserve-router-scheduler-6fdc6df6b6-wr7mm"]
Oct 31 19:52:45.992665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:45.992633    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28d347a1-e182-4a41-9d1f-0ac9ffe842d8" path="/var/lib/kubelet/pods/28d347a1-e182-4a41-9d1f-0ac9ffe842d8/volumes"
Oct 31 19:52:51.740750 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.740710    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"]
Oct 31 19:52:51.741176 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.741163    4484 cpu_manager.go:414] "containerMap: removing container" podUID="28d347a1-e182-4a41-9d1f-0ac9ffe842d8" containerName="main"
Oct 31 19:52:51.741226 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.741177    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d347a1-e182-4a41-9d1f-0ac9ffe842d8" containerName="main"
Oct 31 19:52:51.741276 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.741242    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d347a1-e182-4a41-9d1f-0ac9ffe842d8" containerName="main"
Oct 31 19:52:51.745613 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.745598    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.801595 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.801561    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"]
Oct 31 19:52:51.849084 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.849051    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4e15e5\"/\"default-dockercfg-mk6qv\""
Oct 31 19:52:51.849279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.849123    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4e15e5\"/\"openshift-service-ca.crt\""
Oct 31 19:52:51.849279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.849123    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4e15e5\"/\"kube-root-ca.crt\""
Oct 31 19:52:51.849789 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.849773    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4e15e5\"/\"simulator-single-main-4e15e5-kserve-self-signed-certs\""
Oct 31 19:52:51.865228 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.865200    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tmp-dir\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.865385 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.865240    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tls-certs\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.865385 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.865353    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-home\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.865544 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.865380    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-model-cache\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.865544 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.865513    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xtch\" (UniqueName: \"kubernetes.io/projected/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-kube-api-access-8xtch\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.865661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.865601    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-dshm\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.966938 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.966906    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtch\" (UniqueName: \"kubernetes.io/projected/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-kube-api-access-8xtch\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.967127 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.966965    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-dshm\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.967127 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.967004    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tmp-dir\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.967127 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.967024    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tls-certs\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.967268 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.967154    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-home\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.967268 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.967192    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-model-cache\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.967433 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.967407    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tmp-dir\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.967580 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.967560    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-home\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.967639 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.967568    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-model-cache\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.969465 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.969442    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-dshm\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:51.969752 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:51.969735    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tls-certs\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:52.025326 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:52.025244    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtch\" (UniqueName: \"kubernetes.io/projected/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-kube-api-access-8xtch\") pod \"simulator-single-main-4e15e5-kserve-d6d996b8f-sm879\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") " pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:52.057153 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:52.057122    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:52.430240 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:52.430213    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"]
Oct 31 19:52:52.431810 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:52:52.431779    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f38c0c8_5e8f_4276_9e1d_f664f49086f2.slice/crio-b3a3396299367e3a6675afb277e844db37a1b90b6d012821c46564a3aa10ceba WatchSource:0}: Error finding container b3a3396299367e3a6675afb277e844db37a1b90b6d012821c46564a3aa10ceba: Status 404 returned error can't find the container with id b3a3396299367e3a6675afb277e844db37a1b90b6d012821c46564a3aa10ceba
Oct 31 19:52:53.155438 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:53.155400    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879" event={"ID":"1f38c0c8-5e8f-4276-9e1d-f664f49086f2","Type":"ContainerStarted","Data":"480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41"}
Oct 31 19:52:53.155873 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:53.155450    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879" event={"ID":"1f38c0c8-5e8f-4276-9e1d-f664f49086f2","Type":"ContainerStarted","Data":"b3a3396299367e3a6675afb277e844db37a1b90b6d012821c46564a3aa10ceba"}
Oct 31 19:52:53.296088 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:53.296030    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879" podStartSLOduration=2.296012439 podStartE2EDuration="2.296012439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:53.293616396 +0000 UTC m=+5519.440066259" watchObservedRunningTime="2026-10-31 19:52:53.296012439 +0000 UTC m=+5519.442462306"
Oct 31 19:52:54.058227 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:54.058201    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:52:55.467833 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:52:55.467795    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:52:56.556417 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:52:56.556385    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:53:02.058391 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:02.058353    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:53:02.070775 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:02.070748    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:53:02.211263 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:02.211233    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:53:22.472072 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:22.471982    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:53:25.467977 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:53:25.467939    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:53:37.460372 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.460340    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"]
Oct 31 19:53:37.464141 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.464126    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:37.521861 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.521828    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"]
Oct 31 19:53:37.569726 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.569701    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:53:37.569892 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.569735    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:53:37.614828 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.614793    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e41bc33d-n59jj\" (UID: \"c135cd02-fa93-480d-9096-4ca318a756ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:37.614992 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.614921    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e41bc33d-n59jj\" (UID: \"c135cd02-fa93-480d-9096-4ca318a756ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:37.614992 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.614971    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n58vv\" (UniqueName: \"kubernetes.io/projected/c135cd02-fa93-480d-9096-4ca318a756ba-kube-api-access-n58vv\") pod \"maas-api-revoke-sub-e41bc33d-n59jj\" (UID: \"c135cd02-fa93-480d-9096-4ca318a756ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:37.715758 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.715670    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e41bc33d-n59jj\" (UID: \"c135cd02-fa93-480d-9096-4ca318a756ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:37.715758 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.715714    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n58vv\" (UniqueName: \"kubernetes.io/projected/c135cd02-fa93-480d-9096-4ca318a756ba-kube-api-access-n58vv\") pod \"maas-api-revoke-sub-e41bc33d-n59jj\" (UID: \"c135cd02-fa93-480d-9096-4ca318a756ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:37.715758 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.715760    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e41bc33d-n59jj\" (UID: \"c135cd02-fa93-480d-9096-4ca318a756ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:37.716368 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.716349    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e41bc33d-n59jj\" (UID: \"c135cd02-fa93-480d-9096-4ca318a756ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:37.718257 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.718228    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e41bc33d-n59jj\" (UID: \"c135cd02-fa93-480d-9096-4ca318a756ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:37.775359 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.775328    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n58vv\" (UniqueName: \"kubernetes.io/projected/c135cd02-fa93-480d-9096-4ca318a756ba-kube-api-access-n58vv\") pod \"maas-api-revoke-sub-e41bc33d-n59jj\" (UID: \"c135cd02-fa93-480d-9096-4ca318a756ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:37.776140 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:37.776117    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:38.145144 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:38.145119    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"]
Oct 31 19:53:38.146535 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:53:38.146508    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc135cd02_fa93_480d_9096_4ca318a756ba.slice/crio-e2a513ddb5ca0a476dbdf833f2ba1af4fc21df74491d0f11f4963fed72f30dc7 WatchSource:0}: Error finding container e2a513ddb5ca0a476dbdf833f2ba1af4fc21df74491d0f11f4963fed72f30dc7: Status 404 returned error can't find the container with id e2a513ddb5ca0a476dbdf833f2ba1af4fc21df74491d0f11f4963fed72f30dc7
Oct 31 19:53:38.391663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:38.391631    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj" event={"ID":"c135cd02-fa93-480d-9096-4ca318a756ba","Type":"ContainerStarted","Data":"e2a513ddb5ca0a476dbdf833f2ba1af4fc21df74491d0f11f4963fed72f30dc7"}
Oct 31 19:53:39.397943 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:39.397910    4484 generic.go:356] "Generic (PLEG): container finished" podID="c135cd02-fa93-480d-9096-4ca318a756ba" containerID="2cccf62b95014415267f9526ee68199e2e5eda467becba84ac4a6624c37571b2" exitCode=0
Oct 31 19:53:39.398342 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:39.398004    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj" event={"ID":"c135cd02-fa93-480d-9096-4ca318a756ba","Type":"ContainerDied","Data":"2cccf62b95014415267f9526ee68199e2e5eda467becba84ac4a6624c37571b2"}
Oct 31 19:53:40.536655 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:40.536626    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:40.645200 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:40.645164    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c135cd02-fa93-480d-9096-4ca318a756ba-kube-api-access-n58vv\" (UniqueName: \"kubernetes.io/projected/c135cd02-fa93-480d-9096-4ca318a756ba-kube-api-access-n58vv\") pod \"c135cd02-fa93-480d-9096-4ca318a756ba\" (UID: \"c135cd02-fa93-480d-9096-4ca318a756ba\") "
Oct 31 19:53:40.645400 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:40.645214    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-xks-ca\") pod \"c135cd02-fa93-480d-9096-4ca318a756ba\" (UID: \"c135cd02-fa93-480d-9096-4ca318a756ba\") "
Oct 31 19:53:40.645400 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:40.645255    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-openshift-ca\") pod \"c135cd02-fa93-480d-9096-4ca318a756ba\" (UID: \"c135cd02-fa93-480d-9096-4ca318a756ba\") "
Oct 31 19:53:40.645722 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:40.645700    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-openshift-ca" pod "c135cd02-fa93-480d-9096-4ca318a756ba" (UID: "c135cd02-fa93-480d-9096-4ca318a756ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:53:40.647348 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:40.647323    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c135cd02-fa93-480d-9096-4ca318a756ba-kube-api-access-n58vv" pod "c135cd02-fa93-480d-9096-4ca318a756ba" (UID: "c135cd02-fa93-480d-9096-4ca318a756ba"). InnerVolumeSpecName "kube-api-access-n58vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:40.647437 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:40.647422    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-xks-ca" pod "c135cd02-fa93-480d-9096-4ca318a756ba" (UID: "c135cd02-fa93-480d-9096-4ca318a756ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:53:40.746527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:40.746469    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n58vv\" (UniqueName: \"kubernetes.io/projected/c135cd02-fa93-480d-9096-4ca318a756ba-kube-api-access-n58vv\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:53:40.746527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:40.746521    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:53:40.746527 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:40.746531    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c135cd02-fa93-480d-9096-4ca318a756ba-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:53:41.408235 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:41.408209    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"
Oct 31 19:53:41.408235 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:41.408215    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj" event={"ID":"c135cd02-fa93-480d-9096-4ca318a756ba","Type":"ContainerDied","Data":"e2a513ddb5ca0a476dbdf833f2ba1af4fc21df74491d0f11f4963fed72f30dc7"}
Oct 31 19:53:41.408459 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:41.408258    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2a513ddb5ca0a476dbdf833f2ba1af4fc21df74491d0f11f4963fed72f30dc7"
Oct 31 19:53:47.044335 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:47.044302    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"]
Oct 31 19:53:47.094797 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:47.094762    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-kw2wr"]
Oct 31 19:53:47.992261 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:47.992235    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a277764d-6c36-4e11-8b77-5d39ba95c0c1" path="/var/lib/kubelet/pods/a277764d-6c36-4e11-8b77-5d39ba95c0c1/volumes"
Oct 31 19:53:50.452630 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:50.452600    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:53:53.536953 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:53.536919    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"]
Oct 31 19:53:55.468080 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:53:55.468043    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:53:56.190763 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:53:56.190733    4484 scope.go:122] "RemoveContainer" containerID="b1f7ee859896d5e0b5b2c9c69fdc49481849e63956e6b89565170e41f43367b7"
Oct 31 19:54:00.393318 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:00.393285    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:54:08.557037 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.556978    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879" podUID="1f38c0c8-5e8f-4276-9e1d-f664f49086f2" containerName="main" containerID="cri-o://480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41" gracePeriod=45
Oct 31 19:54:08.811358 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.811300    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:54:08.900227 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.900196    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-kube-api-access-8xtch\" (UniqueName: \"kubernetes.io/projected/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-kube-api-access-8xtch\") pod \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") "
Oct 31 19:54:08.900227 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.900230    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tls-certs\") pod \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") "
Oct 31 19:54:08.900502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.900264    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-home\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-home\") pod \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") "
Oct 31 19:54:08.900502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.900321    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-dshm\") pod \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") "
Oct 31 19:54:08.900502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.900354    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-model-cache\") pod \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") "
Oct 31 19:54:08.900502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.900394    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tmp-dir\") pod \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\" (UID: \"1f38c0c8-5e8f-4276-9e1d-f664f49086f2\") "
Oct 31 19:54:08.900739 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.900662    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-home" pod "1f38c0c8-5e8f-4276-9e1d-f664f49086f2" (UID: "1f38c0c8-5e8f-4276-9e1d-f664f49086f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:54:08.900800 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.900741    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-model-cache" pod "1f38c0c8-5e8f-4276-9e1d-f664f49086f2" (UID: "1f38c0c8-5e8f-4276-9e1d-f664f49086f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:54:08.900800 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.900763    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tmp-dir" pod "1f38c0c8-5e8f-4276-9e1d-f664f49086f2" (UID: "1f38c0c8-5e8f-4276-9e1d-f664f49086f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:54:08.902654 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.902629    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-dshm" pod "1f38c0c8-5e8f-4276-9e1d-f664f49086f2" (UID: "1f38c0c8-5e8f-4276-9e1d-f664f49086f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:54:08.902782 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.902666    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tls-certs" pod "1f38c0c8-5e8f-4276-9e1d-f664f49086f2" (UID: "1f38c0c8-5e8f-4276-9e1d-f664f49086f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:54:08.902782 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:08.902707    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-kube-api-access-8xtch" pod "1f38c0c8-5e8f-4276-9e1d-f664f49086f2" (UID: "1f38c0c8-5e8f-4276-9e1d-f664f49086f2"). InnerVolumeSpecName "kube-api-access-8xtch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:54:09.002052 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.002023    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:54:09.002052 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.002050    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:54:09.002052 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.002059    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:54:09.002290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.002068    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xtch\" (UniqueName: \"kubernetes.io/projected/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-kube-api-access-8xtch\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:54:09.002290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.002077    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:54:09.002290 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.002084    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f38c0c8-5e8f-4276-9e1d-f664f49086f2-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:54:09.546447 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.546411    4484 generic.go:356] "Generic (PLEG): container finished" podID="1f38c0c8-5e8f-4276-9e1d-f664f49086f2" containerID="480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41" exitCode=0
Oct 31 19:54:09.546642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.546459    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"
Oct 31 19:54:09.546642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.546467    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879" event={"ID":"1f38c0c8-5e8f-4276-9e1d-f664f49086f2","Type":"ContainerDied","Data":"480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41"}
Oct 31 19:54:09.546642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.546523    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879" event={"ID":"1f38c0c8-5e8f-4276-9e1d-f664f49086f2","Type":"ContainerDied","Data":"b3a3396299367e3a6675afb277e844db37a1b90b6d012821c46564a3aa10ceba"}
Oct 31 19:54:09.546642 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.546546    4484 scope.go:122] "RemoveContainer" containerID="480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41"
Oct 31 19:54:09.555834 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.555812    4484 scope.go:122] "RemoveContainer" containerID="480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41"
Oct 31 19:54:09.556101 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:54:09.556080    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41\": container with ID starting with 480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41 not found: ID does not exist" containerID="480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41"
Oct 31 19:54:09.556152 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.556108    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41"} err="failed to get container status \"480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41\": rpc error: code = NotFound desc = could not find container \"480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41\": container with ID starting with 480b0daf0c5f1c59a33326d53bce5c2e0d62040a0795a5b4b4977fe7ef25ac41 not found: ID does not exist"
Oct 31 19:54:09.715591 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.715556    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"]
Oct 31 19:54:09.718957 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.718921    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-4e15e5/simulator-single-main-4e15e5-kserve-d6d996b8f-sm879"]
Oct 31 19:54:09.992245 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:09.992220    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f38c0c8-5e8f-4276-9e1d-f664f49086f2" path="/var/lib/kubelet/pods/1f38c0c8-5e8f-4276-9e1d-f664f49086f2/volumes"
Oct 31 19:54:25.466930 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:54:25.466896    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:54:32.459672 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.459638    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"]
Oct 31 19:54:32.460150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.460042    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c135cd02-fa93-480d-9096-4ca318a756ba" containerName="revoke-keys"
Oct 31 19:54:32.460150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.460056    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c135cd02-fa93-480d-9096-4ca318a756ba" containerName="revoke-keys"
Oct 31 19:54:32.460150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.460075    4484 cpu_manager.go:414] "containerMap: removing container" podUID="1f38c0c8-5e8f-4276-9e1d-f664f49086f2" containerName="main"
Oct 31 19:54:32.460150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.460080    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f38c0c8-5e8f-4276-9e1d-f664f49086f2" containerName="main"
Oct 31 19:54:32.460150 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.460151    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f38c0c8-5e8f-4276-9e1d-f664f49086f2" containerName="main"
Oct 31 19:54:32.460317 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.460164    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c135cd02-fa93-480d-9096-4ca318a756ba" containerName="revoke-keys"
Oct 31 19:54:32.463422 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.463403    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:32.530464 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.530430    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q92n\" (UniqueName: \"kubernetes.io/projected/c5d6a110-6841-4770-a604-ed3db0816c24-kube-api-access-5q92n\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:32.530602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.530549    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:32.530602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.530570    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:32.530691 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.530655    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d6a110-6841-4770-a604-ed3db0816c24-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:32.584444 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.584414    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4e15e5-serving-cert\""
Oct 31 19:54:32.584444 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.584420    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 19:54:32.584665 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.584450    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czkfc\""
Oct 31 19:54:32.586825 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.586807    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"]
Oct 31 19:54:32.632125 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.632097    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d6a110-6841-4770-a604-ed3db0816c24-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:32.632293 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.632159    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q92n\" (UniqueName: \"kubernetes.io/projected/c5d6a110-6841-4770-a604-ed3db0816c24-kube-api-access-5q92n\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:32.632293 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.632208    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:32.632293 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.632230    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:32.632426 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:54:32.632366    4484 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-serving-cert: secret "maas-api-e2e-worker-single-main-4e15e5-serving-cert" not found
Oct 31 19:54:32.632494 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:54:32.632440    4484 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-maas-api-tls podName:c5d6a110-6841-4770-a604-ed3db0816c24 nodeName:}" failed. No retries permitted until 2026-10-31 19:54:33.132410097 +0000 UTC m=+5619.278859965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-maas-api-tls") pod "maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr" (UID: "c5d6a110-6841-4770-a604-ed3db0816c24") : secret "maas-api-e2e-worker-single-main-4e15e5-serving-cert" not found
Oct 31 19:54:32.633060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.632981    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d6a110-6841-4770-a604-ed3db0816c24-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:32.634777 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.634755    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:32.700187 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:32.700160    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q92n\" (UniqueName: \"kubernetes.io/projected/c5d6a110-6841-4770-a604-ed3db0816c24-kube-api-access-5q92n\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:33.137985 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:33.137944    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:33.140619 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:33.140588    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:33.376387 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:33.376357    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:33.766096 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:33.766067    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"]
Oct 31 19:54:33.767297 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:54:33.767272    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d6a110_6841_4770_a604_ed3db0816c24.slice/crio-4db4ce88074362a39b8a77aba3d249819e441620f45694134b5e97bce4cfd505 WatchSource:0}: Error finding container 4db4ce88074362a39b8a77aba3d249819e441620f45694134b5e97bce4cfd505: Status 404 returned error can't find the container with id 4db4ce88074362a39b8a77aba3d249819e441620f45694134b5e97bce4cfd505
Oct 31 19:54:33.768468 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:33.768451    4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:54:34.050548 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.050464    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"]
Oct 31 19:54:34.055357 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.055339    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"
Oct 31 19:54:34.113194 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.113163    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-2df17b12-dockercfg-tf7dk\""
Oct 31 19:54:34.115399 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.115371    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"]
Oct 31 19:54:34.147238 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.147207    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g248x\" (UniqueName: \"kubernetes.io/projected/c1b9742c-9434-455a-8cc2-13dcad588622-kube-api-access-g248x\") pod \"payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj\" (UID: \"c1b9742c-9434-455a-8cc2-13dcad588622\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"
Oct 31 19:54:34.147442 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.147398    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b9742c-9434-455a-8cc2-13dcad588622-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj\" (UID: \"c1b9742c-9434-455a-8cc2-13dcad588622\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"
Oct 31 19:54:34.189809 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.189778    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4e15e5\""
Oct 31 19:54:34.204556 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.204523    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"]
Oct 31 19:54:34.208125 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.208110    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"
Oct 31 19:54:34.247942 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.247916    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hq4p\" (UniqueName: \"kubernetes.io/projected/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-kube-api-access-9hq4p\") pod \"payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck\" (UID: \"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"
Oct 31 19:54:34.248073 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.247971    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b9742c-9434-455a-8cc2-13dcad588622-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj\" (UID: \"c1b9742c-9434-455a-8cc2-13dcad588622\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"
Oct 31 19:54:34.248125 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.248064    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g248x\" (UniqueName: \"kubernetes.io/projected/c1b9742c-9434-455a-8cc2-13dcad588622-kube-api-access-g248x\") pod \"payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj\" (UID: \"c1b9742c-9434-455a-8cc2-13dcad588622\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"
Oct 31 19:54:34.248162 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.248123    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck\" (UID: \"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"
Oct 31 19:54:34.248710 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.248693    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b9742c-9434-455a-8cc2-13dcad588622-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj\" (UID: \"c1b9742c-9434-455a-8cc2-13dcad588622\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"
Oct 31 19:54:34.272007 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.271986    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"]
Oct 31 19:54:34.307791 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.307728    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g248x\" (UniqueName: \"kubernetes.io/projected/c1b9742c-9434-455a-8cc2-13dcad588622-kube-api-access-g248x\") pod \"payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj\" (UID: \"c1b9742c-9434-455a-8cc2-13dcad588622\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"
Oct 31 19:54:34.348997 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.348960    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck\" (UID: \"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"
Oct 31 19:54:34.349164 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.349056    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hq4p\" (UniqueName: \"kubernetes.io/projected/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-kube-api-access-9hq4p\") pod \"payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck\" (UID: \"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"
Oct 31 19:54:34.349724 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.349707    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck\" (UID: \"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"
Oct 31 19:54:34.368587 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.368562    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"
Oct 31 19:54:34.413206 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.413168    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hq4p\" (UniqueName: \"kubernetes.io/projected/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-kube-api-access-9hq4p\") pod \"payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck\" (UID: \"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"
Oct 31 19:54:34.521464 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.521435    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"
Oct 31 19:54:34.670282 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.670249    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr" event={"ID":"c5d6a110-6841-4770-a604-ed3db0816c24","Type":"ContainerStarted","Data":"4db4ce88074362a39b8a77aba3d249819e441620f45694134b5e97bce4cfd505"}
Oct 31 19:54:34.753627 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:54:34.753588    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b9742c_9434_455a_8cc2_13dcad588622.slice/crio-df6031edba9b5c650302caaae056deed9a5826a4978f7647859136369656b8c1 WatchSource:0}: Error finding container df6031edba9b5c650302caaae056deed9a5826a4978f7647859136369656b8c1: Status 404 returned error can't find the container with id df6031edba9b5c650302caaae056deed9a5826a4978f7647859136369656b8c1
Oct 31 19:54:34.754648 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.754626    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"]
Oct 31 19:54:34.896959 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:34.896932    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"]
Oct 31 19:54:34.898252 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:54:34.898226    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf91d1e8_ddfc_44dc_a184_064f70dbf6bc.slice/crio-38109ea3c00792ef56bd919e3a1082088c7e6b466bb4e78a5728cc09ebafb82a WatchSource:0}: Error finding container 38109ea3c00792ef56bd919e3a1082088c7e6b466bb4e78a5728cc09ebafb82a: Status 404 returned error can't find the container with id 38109ea3c00792ef56bd919e3a1082088c7e6b466bb4e78a5728cc09ebafb82a
Oct 31 19:54:35.075310 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.075233    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"]
Oct 31 19:54:35.078845 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.078823    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-pn286"]
Oct 31 19:54:35.677385 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.677299    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr" event={"ID":"c5d6a110-6841-4770-a604-ed3db0816c24","Type":"ContainerStarted","Data":"f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0"}
Oct 31 19:54:35.677822 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.677786    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:35.679198 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.679168    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck" event={"ID":"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc","Type":"ContainerStarted","Data":"6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad"}
Oct 31 19:54:35.679361 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.679217    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck" event={"ID":"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc","Type":"ContainerStarted","Data":"38109ea3c00792ef56bd919e3a1082088c7e6b466bb4e78a5728cc09ebafb82a"}
Oct 31 19:54:35.679361 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.679241    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"
Oct 31 19:54:35.680886 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.680862    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj" event={"ID":"c1b9742c-9434-455a-8cc2-13dcad588622","Type":"ContainerStarted","Data":"25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17"}
Oct 31 19:54:35.681010 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.680907    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj" event={"ID":"c1b9742c-9434-455a-8cc2-13dcad588622","Type":"ContainerStarted","Data":"df6031edba9b5c650302caaae056deed9a5826a4978f7647859136369656b8c1"}
Oct 31 19:54:35.681010 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.680954    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"
Oct 31 19:54:35.801189 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.801137    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr" podStartSLOduration=2.932911521 podStartE2EDuration="3.801125662s" totalImagesPullingTime="868.214141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:35.799301283 +0000 UTC m=+5621.945751160" watchObservedRunningTime="2026-10-31 19:54:35.801125662 +0000 UTC m=+5621.947575527"
Oct 31 19:54:35.924667 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.924607    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck" podStartSLOduration=1.924588071 podStartE2EDuration="1.924588071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:35.92138239 +0000 UTC m=+5622.067832256" watchObservedRunningTime="2026-10-31 19:54:35.924588071 +0000 UTC m=+5622.071037936"
Oct 31 19:54:35.993463 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:35.993431    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ee7d45-c55b-4db6-9c9f-a8b239350cd9" path="/var/lib/kubelet/pods/e5ee7d45-c55b-4db6-9c9f-a8b239350cd9/volumes"
Oct 31 19:54:36.044745 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:36.044695    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj" podStartSLOduration=2.044680985 podStartE2EDuration="2.044680985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:36.044088297 +0000 UTC m=+5622.190538162" watchObservedRunningTime="2026-10-31 19:54:36.044680985 +0000 UTC m=+5622.191130848"
Oct 31 19:54:41.691293 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:41.691262    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:54:42.584855 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:42.584819    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:54:46.688139 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:46.688108    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"
Oct 31 19:54:46.688595 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:46.688195    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"
Oct 31 19:54:55.467030 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:54:55.466991    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:54:56.216354 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:56.216325    4484 scope.go:122] "RemoveContainer" containerID="96c16528f50b8555493d84ce4d203a9db35ca32484597e776e7370192c1c6e94"
Oct 31 19:54:59.032126 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:59.032087    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"]
Oct 31 19:54:59.141978 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:54:59.141946    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"]
Oct 31 19:55:02.339101 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.339064    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"]
Oct 31 19:55:02.339596 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.339276    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr" podUID="c5d6a110-6841-4770-a604-ed3db0816c24" containerName="maas-api" containerID="cri-o://f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0" gracePeriod=30
Oct 31 19:55:02.590972 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.590913    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:55:02.717446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.717410    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d6a110-6841-4770-a604-ed3db0816c24-kube-api-access-5q92n\" (UniqueName: \"kubernetes.io/projected/c5d6a110-6841-4770-a604-ed3db0816c24-kube-api-access-5q92n\") pod \"c5d6a110-6841-4770-a604-ed3db0816c24\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") "
Oct 31 19:55:02.717446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.717542    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-maas-api-tls\") pod \"c5d6a110-6841-4770-a604-ed3db0816c24\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") "
Oct 31 19:55:02.717446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.717607    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-metrics-certs\") pod \"c5d6a110-6841-4770-a604-ed3db0816c24\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") "
Oct 31 19:55:02.717446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.717699    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d6a110-6841-4770-a604-ed3db0816c24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d6a110-6841-4770-a604-ed3db0816c24-openshift-service-ca\") pod \"c5d6a110-6841-4770-a604-ed3db0816c24\" (UID: \"c5d6a110-6841-4770-a604-ed3db0816c24\") "
Oct 31 19:55:02.717446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.718296    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d6a110-6841-4770-a604-ed3db0816c24-openshift-service-ca" pod "c5d6a110-6841-4770-a604-ed3db0816c24" (UID: "c5d6a110-6841-4770-a604-ed3db0816c24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:55:02.717446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.721423    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d6a110-6841-4770-a604-ed3db0816c24-kube-api-access-5q92n" pod "c5d6a110-6841-4770-a604-ed3db0816c24" (UID: "c5d6a110-6841-4770-a604-ed3db0816c24"). InnerVolumeSpecName "kube-api-access-5q92n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:55:02.717446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.723718    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-maas-api-tls" pod "c5d6a110-6841-4770-a604-ed3db0816c24" (UID: "c5d6a110-6841-4770-a604-ed3db0816c24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:55:02.717446 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.723863    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-metrics-certs" pod "c5d6a110-6841-4770-a604-ed3db0816c24" (UID: "c5d6a110-6841-4770-a604-ed3db0816c24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:55:02.814682 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.814649    4484 generic.go:356] "Generic (PLEG): container finished" podID="c5d6a110-6841-4770-a604-ed3db0816c24" containerID="f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0" exitCode=0
Oct 31 19:55:02.814863 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.814694    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"
Oct 31 19:55:02.814863 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.814725    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr" event={"ID":"c5d6a110-6841-4770-a604-ed3db0816c24","Type":"ContainerDied","Data":"f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0"}
Oct 31 19:55:02.814863 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.814765    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr" event={"ID":"c5d6a110-6841-4770-a604-ed3db0816c24","Type":"ContainerDied","Data":"4db4ce88074362a39b8a77aba3d249819e441620f45694134b5e97bce4cfd505"}
Oct 31 19:55:02.814863 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.814788    4484 scope.go:122] "RemoveContainer" containerID="f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0"
Oct 31 19:55:02.819004 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.818974    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5d6a110-6841-4770-a604-ed3db0816c24-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:02.819120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.819007    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q92n\" (UniqueName: \"kubernetes.io/projected/c5d6a110-6841-4770-a604-ed3db0816c24-kube-api-access-5q92n\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:02.819120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.819022    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:02.819120 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.819034    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d6a110-6841-4770-a604-ed3db0816c24-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:02.824270 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.824251    4484 scope.go:122] "RemoveContainer" containerID="f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0"
Oct 31 19:55:02.824546 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:55:02.824530    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0\": container with ID starting with f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0 not found: ID does not exist" containerID="f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0"
Oct 31 19:55:02.824613 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.824555    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0"} err="failed to get container status \"f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0\": rpc error: code = NotFound desc = could not find container \"f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0\": container with ID starting with f866744e1e51d69fb7f06c372b17c8241ddb59c87d136b5bf99c570138a664f0 not found: ID does not exist"
Oct 31 19:55:02.985877 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.985798    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"]
Oct 31 19:55:02.992157 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:02.992133    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4e15e5-7b86875d66-kdhnr"]
Oct 31 19:55:03.992343 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:03.992308    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d6a110-6841-4770-a604-ed3db0816c24" path="/var/lib/kubelet/pods/c5d6a110-6841-4770-a604-ed3db0816c24/volumes"
Oct 31 19:55:04.032620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.032587    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck" podUID="cf91d1e8-ddfc-44dc-a184-064f70dbf6bc" containerName="payload-processing" containerID="cri-o://6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad" gracePeriod=25
Oct 31 19:55:04.142965 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.142925    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj" podUID="c1b9742c-9434-455a-8cc2-13dcad588622" containerName="payload-pre-processing" containerID="cri-o://25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17" gracePeriod=25
Oct 31 19:55:04.289688 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.288984    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"
Oct 31 19:55:04.400833 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.400812    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"
Oct 31 19:55:04.433952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.433918    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-kube-api-access-9hq4p\" (UniqueName: \"kubernetes.io/projected/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-kube-api-access-9hq4p\") pod \"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc\" (UID: \"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc\") "
Oct 31 19:55:04.434119 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.433962    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-plugins-config-volume\") pod \"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc\" (UID: \"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc\") "
Oct 31 19:55:04.434525 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.434457    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-plugins-config-volume" pod "cf91d1e8-ddfc-44dc-a184-064f70dbf6bc" (UID: "cf91d1e8-ddfc-44dc-a184-064f70dbf6bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:55:04.436399 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.436368    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-kube-api-access-9hq4p" pod "cf91d1e8-ddfc-44dc-a184-064f70dbf6bc" (UID: "cf91d1e8-ddfc-44dc-a184-064f70dbf6bc"). InnerVolumeSpecName "kube-api-access-9hq4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:55:04.535099 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.535031    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b9742c-9434-455a-8cc2-13dcad588622-kube-api-access-g248x\" (UniqueName: \"kubernetes.io/projected/c1b9742c-9434-455a-8cc2-13dcad588622-kube-api-access-g248x\") pod \"c1b9742c-9434-455a-8cc2-13dcad588622\" (UID: \"c1b9742c-9434-455a-8cc2-13dcad588622\") "
Oct 31 19:55:04.535099 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.535077    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b9742c-9434-455a-8cc2-13dcad588622-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b9742c-9434-455a-8cc2-13dcad588622-plugins-config-volume\") pod \"c1b9742c-9434-455a-8cc2-13dcad588622\" (UID: \"c1b9742c-9434-455a-8cc2-13dcad588622\") "
Oct 31 19:55:04.535362 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.535344    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hq4p\" (UniqueName: \"kubernetes.io/projected/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-kube-api-access-9hq4p\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:04.535439 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.535368    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:04.535543 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.535521    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b9742c-9434-455a-8cc2-13dcad588622-plugins-config-volume" pod "c1b9742c-9434-455a-8cc2-13dcad588622" (UID: "c1b9742c-9434-455a-8cc2-13dcad588622"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:55:04.537428 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.537402    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b9742c-9434-455a-8cc2-13dcad588622-kube-api-access-g248x" pod "c1b9742c-9434-455a-8cc2-13dcad588622" (UID: "c1b9742c-9434-455a-8cc2-13dcad588622"). InnerVolumeSpecName "kube-api-access-g248x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:55:04.636780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.636751    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b9742c-9434-455a-8cc2-13dcad588622-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:04.636780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.636777    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g248x\" (UniqueName: \"kubernetes.io/projected/c1b9742c-9434-455a-8cc2-13dcad588622-kube-api-access-g248x\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:55:04.827017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.826931    4484 generic.go:356] "Generic (PLEG): container finished" podID="cf91d1e8-ddfc-44dc-a184-064f70dbf6bc" containerID="6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad" exitCode=0
Oct 31 19:55:04.827017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.826974    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"
Oct 31 19:55:04.827261 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.827027    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck" event={"ID":"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc","Type":"ContainerDied","Data":"6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad"}
Oct 31 19:55:04.827261 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.827083    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck" event={"ID":"cf91d1e8-ddfc-44dc-a184-064f70dbf6bc","Type":"ContainerDied","Data":"38109ea3c00792ef56bd919e3a1082088c7e6b466bb4e78a5728cc09ebafb82a"}
Oct 31 19:55:04.827261 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.827116    4484 scope.go:122] "RemoveContainer" containerID="6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad"
Oct 31 19:55:04.828870 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.828837    4484 generic.go:356] "Generic (PLEG): container finished" podID="c1b9742c-9434-455a-8cc2-13dcad588622" containerID="25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17" exitCode=0
Oct 31 19:55:04.829017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.828890    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"
Oct 31 19:55:04.829017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.828923    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj" event={"ID":"c1b9742c-9434-455a-8cc2-13dcad588622","Type":"ContainerDied","Data":"25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17"}
Oct 31 19:55:04.829017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.828993    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj" event={"ID":"c1b9742c-9434-455a-8cc2-13dcad588622","Type":"ContainerDied","Data":"df6031edba9b5c650302caaae056deed9a5826a4978f7647859136369656b8c1"}
Oct 31 19:55:04.839983 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.839965    4484 scope.go:122] "RemoveContainer" containerID="6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad"
Oct 31 19:55:04.840281 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:55:04.840256    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad\": container with ID starting with 6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad not found: ID does not exist" containerID="6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad"
Oct 31 19:55:04.840365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.840290    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad"} err="failed to get container status \"6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad\": rpc error: code = NotFound desc = could not find container \"6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad\": container with ID starting with 6cfaea157b58b35440aae27d570f4bc57339f53071f56c71c9c54dbfc10000ad not found: ID does not exist"
Oct 31 19:55:04.840365 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.840310    4484 scope.go:122] "RemoveContainer" containerID="25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17"
Oct 31 19:55:04.848194 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.848177    4484 scope.go:122] "RemoveContainer" containerID="25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17"
Oct 31 19:55:04.848435 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:55:04.848418    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17\": container with ID starting with 25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17 not found: ID does not exist" containerID="25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17"
Oct 31 19:55:04.848530 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.848439    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17"} err="failed to get container status \"25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17\": rpc error: code = NotFound desc = could not find container \"25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17\": container with ID starting with 25d06aab0d4bed68ea4a648d95a17dbe23f5b37703446ee4d03b26a5cc00af17 not found: ID does not exist"
Oct 31 19:55:04.998188 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:04.998154    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"]
Oct 31 19:55:05.001149 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:05.001125    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e15e5-84444cd99m8gck"]
Oct 31 19:55:05.120184 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:05.120102    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"]
Oct 31 19:55:05.130845 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:05.130804    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4e15e5-665d54bsdj"]
Oct 31 19:55:05.994017 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:05.993985    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b9742c-9434-455a-8cc2-13dcad588622" path="/var/lib/kubelet/pods/c1b9742c-9434-455a-8cc2-13dcad588622/volumes"
Oct 31 19:55:05.994460 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:05.994439    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf91d1e8-ddfc-44dc-a184-064f70dbf6bc" path="/var/lib/kubelet/pods/cf91d1e8-ddfc-44dc-a184-064f70dbf6bc/volumes"
Oct 31 19:55:06.900586 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:06.900551    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:55:08.059826 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:08.059785    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"]
Oct 31 19:55:08.110885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:08.110855    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mqwmj"]
Oct 31 19:55:09.992112 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:09.992077    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1805861e-833d-415d-82f1-bdb8aac7d5c3" path="/var/lib/kubelet/pods/1805861e-833d-415d-82f1-bdb8aac7d5c3/volumes"
Oct 31 19:55:17.000978 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:17.000943    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:55:25.467102 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:55:25.467065    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:55:28.047976 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:28.047943    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"]
Oct 31 19:55:28.098894 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:28.098865    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s7lrm"]
Oct 31 19:55:29.992491 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:29.992449    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d80a47f-8692-434d-800d-b9ebc97dea05" path="/var/lib/kubelet/pods/9d80a47f-8692-434d-800d-b9ebc97dea05/volumes"
Oct 31 19:55:40.046588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.046554    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"]
Oct 31 19:55:40.097512 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.097483    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xh8r5"]
Oct 31 19:55:40.419183 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.419106    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"]
Oct 31 19:55:40.419574 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.419560    4484 cpu_manager.go:414] "containerMap: removing container" podUID="cf91d1e8-ddfc-44dc-a184-064f70dbf6bc" containerName="payload-processing"
Oct 31 19:55:40.419622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.419576    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf91d1e8-ddfc-44dc-a184-064f70dbf6bc" containerName="payload-processing"
Oct 31 19:55:40.419622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.419590    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c5d6a110-6841-4770-a604-ed3db0816c24" containerName="maas-api"
Oct 31 19:55:40.419622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.419595    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d6a110-6841-4770-a604-ed3db0816c24" containerName="maas-api"
Oct 31 19:55:40.419622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.419608    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c1b9742c-9434-455a-8cc2-13dcad588622" containerName="payload-pre-processing"
Oct 31 19:55:40.419622 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.419614    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b9742c-9434-455a-8cc2-13dcad588622" containerName="payload-pre-processing"
Oct 31 19:55:40.419789 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.419691    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf91d1e8-ddfc-44dc-a184-064f70dbf6bc" containerName="payload-processing"
Oct 31 19:55:40.419789 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.419698    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d6a110-6841-4770-a604-ed3db0816c24" containerName="maas-api"
Oct 31 19:55:40.419789 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.419705    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b9742c-9434-455a-8cc2-13dcad588622" containerName="payload-pre-processing"
Oct 31 19:55:40.422703 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.422687    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.516271 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.516239    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-home\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.516271 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.516274    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-model-cache\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.516520 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.516398    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pl4b\" (UniqueName: \"kubernetes.io/projected/c603fe2e-2417-4088-945a-199a01ca550b-kube-api-access-4pl4b\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.516520 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.516426    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-dshm\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.516520 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.516467    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c603fe2e-2417-4088-945a-199a01ca550b-tls-certs\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.516628 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.516549    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-tmp-dir\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.529112 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.529087    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-20b73ef4-kserve-self-signed-certs\""
Oct 31 19:55:40.531443 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.531424    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"]
Oct 31 19:55:40.618005 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.617964    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-home\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.618005 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.618014    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-model-cache\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.618288 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.618104    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pl4b\" (UniqueName: \"kubernetes.io/projected/c603fe2e-2417-4088-945a-199a01ca550b-kube-api-access-4pl4b\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.618288 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.618163    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-dshm\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.618288 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.618228    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c603fe2e-2417-4088-945a-199a01ca550b-tls-certs\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.618455 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.618302    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-tmp-dir\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.618455 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.618359    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-home\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.618455 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.618392    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-model-cache\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.618701 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.618658    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-tmp-dir\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.620522 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.620492    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-dshm\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.620894 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.620874    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c603fe2e-2417-4088-945a-199a01ca550b-tls-certs\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.680324 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.680266    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pl4b\" (UniqueName: \"kubernetes.io/projected/c603fe2e-2417-4088-945a-199a01ca550b-kube-api-access-4pl4b\") pod \"e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") " pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:40.734402 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:40.734370    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:41.109774 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:41.109465    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"]
Oct 31 19:55:41.111390 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:55:41.111354    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc603fe2e_2417_4088_945a_199a01ca550b.slice/crio-f6e826dfffd1ea904833a227eb9cbe3f4ed6a8d05f59fc5bbc254c840887f560 WatchSource:0}: Error finding container f6e826dfffd1ea904833a227eb9cbe3f4ed6a8d05f59fc5bbc254c840887f560: Status 404 returned error can't find the container with id f6e826dfffd1ea904833a227eb9cbe3f4ed6a8d05f59fc5bbc254c840887f560
Oct 31 19:55:41.992796 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:41.992763    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2872a3-4f1d-4c5b-9937-8ccc8777eebe" path="/var/lib/kubelet/pods/9c2872a3-4f1d-4c5b-9937-8ccc8777eebe/volumes"
Oct 31 19:55:42.028208 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:42.028178    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q" event={"ID":"c603fe2e-2417-4088-945a-199a01ca550b","Type":"ContainerStarted","Data":"b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a"}
Oct 31 19:55:42.028208 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:42.028213    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q" event={"ID":"c603fe2e-2417-4088-945a-199a01ca550b","Type":"ContainerStarted","Data":"f6e826dfffd1ea904833a227eb9cbe3f4ed6a8d05f59fc5bbc254c840887f560"}
Oct 31 19:55:42.147457 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:42.147410    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q" podStartSLOduration=2.147396861 podStartE2EDuration="2.147396861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:42.146701637 +0000 UTC m=+5688.293151504" watchObservedRunningTime="2026-10-31 19:55:42.147396861 +0000 UTC m=+5688.293846724"
Oct 31 19:55:42.735125 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:42.735092    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:45.066616 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:45.066577    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:55:50.735177 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:50.735140    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:50.747279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:50.747257    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:51.083769 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:51.083735    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:55:55.016131 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:55.016103    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:55:55.022952 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:55.022933    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 19:55:55.467756 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:55:55.467670    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:55:56.240783 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:55:56.240759    4484 scope.go:122] "RemoveContainer" containerID="b6872b5782e149c2fd3894e2cc0d4d7c97d8486c4b325d59ffc66fe8c0040f5b"
Oct 31 19:56:08.537443 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.537364    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"]
Oct 31 19:56:08.542366 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.542348    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:08.598095 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.598064    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"]
Oct 31 19:56:08.635403 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.635371    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62pms\" (UniqueName: \"kubernetes.io/projected/48d92140-c003-46e2-9717-9ddc11015581-kube-api-access-62pms\") pod \"maas-api-revoke-sub-452b6d7b-wfn7l\" (UID: \"48d92140-c003-46e2-9717-9ddc11015581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:08.635601 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.635437    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d92140-c003-46e2-9717-9ddc11015581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wfn7l\" (UID: \"48d92140-c003-46e2-9717-9ddc11015581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:08.635601 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.635466    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d92140-c003-46e2-9717-9ddc11015581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wfn7l\" (UID: \"48d92140-c003-46e2-9717-9ddc11015581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:08.646297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.646274    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:56:08.646584 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.646568    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:56:08.736012 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.735985    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d92140-c003-46e2-9717-9ddc11015581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wfn7l\" (UID: \"48d92140-c003-46e2-9717-9ddc11015581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:08.736139 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.736018    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d92140-c003-46e2-9717-9ddc11015581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wfn7l\" (UID: \"48d92140-c003-46e2-9717-9ddc11015581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:08.736139 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.736094    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62pms\" (UniqueName: \"kubernetes.io/projected/48d92140-c003-46e2-9717-9ddc11015581-kube-api-access-62pms\") pod \"maas-api-revoke-sub-452b6d7b-wfn7l\" (UID: \"48d92140-c003-46e2-9717-9ddc11015581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:08.736702 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.736674    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d92140-c003-46e2-9717-9ddc11015581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wfn7l\" (UID: \"48d92140-c003-46e2-9717-9ddc11015581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:08.738586 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.738561    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d92140-c003-46e2-9717-9ddc11015581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wfn7l\" (UID: \"48d92140-c003-46e2-9717-9ddc11015581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:08.794821 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.794758    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62pms\" (UniqueName: \"kubernetes.io/projected/48d92140-c003-46e2-9717-9ddc11015581-kube-api-access-62pms\") pod \"maas-api-revoke-sub-452b6d7b-wfn7l\" (UID: \"48d92140-c003-46e2-9717-9ddc11015581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:08.854233 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:08.854204    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:09.224921 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:09.224893    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"]
Oct 31 19:56:09.226181 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:56:09.226156    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d92140_c003_46e2_9717_9ddc11015581.slice/crio-084e5f653fad95b3fcfae0302514c1fb3f6b4df57541ad503b46b64ebb1572d8 WatchSource:0}: Error finding container 084e5f653fad95b3fcfae0302514c1fb3f6b4df57541ad503b46b64ebb1572d8: Status 404 returned error can't find the container with id 084e5f653fad95b3fcfae0302514c1fb3f6b4df57541ad503b46b64ebb1572d8
Oct 31 19:56:09.886771 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:09.886736    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:56:10.166655 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:10.166577    4484 generic.go:356] "Generic (PLEG): container finished" podID="48d92140-c003-46e2-9717-9ddc11015581" containerID="639d4f4f9e2d0bf391779a759f5933c88cab4401922b3ee2ddd3107acd08fb95" exitCode=0
Oct 31 19:56:10.166801 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:10.166657    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l" event={"ID":"48d92140-c003-46e2-9717-9ddc11015581","Type":"ContainerDied","Data":"639d4f4f9e2d0bf391779a759f5933c88cab4401922b3ee2ddd3107acd08fb95"}
Oct 31 19:56:10.166801 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:10.166708    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l" event={"ID":"48d92140-c003-46e2-9717-9ddc11015581","Type":"ContainerStarted","Data":"084e5f653fad95b3fcfae0302514c1fb3f6b4df57541ad503b46b64ebb1572d8"}
Oct 31 19:56:11.449230 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:11.449164    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:11.583794 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:11.583758    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48d92140-c003-46e2-9717-9ddc11015581-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d92140-c003-46e2-9717-9ddc11015581-maas-api-xks-ca\") pod \"48d92140-c003-46e2-9717-9ddc11015581\" (UID: \"48d92140-c003-46e2-9717-9ddc11015581\") "
Oct 31 19:56:11.584599 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:11.584575    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d92140-c003-46e2-9717-9ddc11015581-kube-api-access-62pms\" (UniqueName: \"kubernetes.io/projected/48d92140-c003-46e2-9717-9ddc11015581-kube-api-access-62pms\") pod \"48d92140-c003-46e2-9717-9ddc11015581\" (UID: \"48d92140-c003-46e2-9717-9ddc11015581\") "
Oct 31 19:56:11.585010 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:11.584992    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d92140-c003-46e2-9717-9ddc11015581-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d92140-c003-46e2-9717-9ddc11015581-maas-api-openshift-ca\") pod \"48d92140-c003-46e2-9717-9ddc11015581\" (UID: \"48d92140-c003-46e2-9717-9ddc11015581\") "
Oct 31 19:56:11.585818 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:11.585796    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d92140-c003-46e2-9717-9ddc11015581-maas-api-openshift-ca" pod "48d92140-c003-46e2-9717-9ddc11015581" (UID: "48d92140-c003-46e2-9717-9ddc11015581"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:11.605696 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:11.605644    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d92140-c003-46e2-9717-9ddc11015581-maas-api-xks-ca" pod "48d92140-c003-46e2-9717-9ddc11015581" (UID: "48d92140-c003-46e2-9717-9ddc11015581"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:56:11.611677 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:11.611645    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d92140-c003-46e2-9717-9ddc11015581-kube-api-access-62pms" pod "48d92140-c003-46e2-9717-9ddc11015581" (UID: "48d92140-c003-46e2-9717-9ddc11015581"). InnerVolumeSpecName "kube-api-access-62pms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:11.695598 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:11.695294    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62pms\" (UniqueName: \"kubernetes.io/projected/48d92140-c003-46e2-9717-9ddc11015581-kube-api-access-62pms\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:11.695598 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:11.695332    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d92140-c003-46e2-9717-9ddc11015581-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:11.695598 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:11.695349    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d92140-c003-46e2-9717-9ddc11015581-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:12.208181 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:12.208147    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l" event={"ID":"48d92140-c003-46e2-9717-9ddc11015581","Type":"ContainerDied","Data":"084e5f653fad95b3fcfae0302514c1fb3f6b4df57541ad503b46b64ebb1572d8"}
Oct 31 19:56:12.208662 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:12.208646    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="084e5f653fad95b3fcfae0302514c1fb3f6b4df57541ad503b46b64ebb1572d8"
Oct 31 19:56:12.208811 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:12.208455    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"
Oct 31 19:56:25.467591 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:56:25.467557    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:56:25.573968 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:25.573933    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:56:33.771972 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:33.771937    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"]
Oct 31 19:56:38.986308 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:38.986278    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:56:48.791926 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:48.791889    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q" podUID="c603fe2e-2417-4088-945a-199a01ca550b" containerName="main" containerID="cri-o://b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a" gracePeriod=45
Oct 31 19:56:49.047588 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.047532    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:56:49.208087 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.208056    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-home\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-home\") pod \"c603fe2e-2417-4088-945a-199a01ca550b\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") "
Oct 31 19:56:49.208289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.208163    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c603fe2e-2417-4088-945a-199a01ca550b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c603fe2e-2417-4088-945a-199a01ca550b-tls-certs\") pod \"c603fe2e-2417-4088-945a-199a01ca550b\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") "
Oct 31 19:56:49.208289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.208188    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-tmp-dir\") pod \"c603fe2e-2417-4088-945a-199a01ca550b\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") "
Oct 31 19:56:49.208289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.208208    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-model-cache\") pod \"c603fe2e-2417-4088-945a-199a01ca550b\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") "
Oct 31 19:56:49.208289 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.208254    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c603fe2e-2417-4088-945a-199a01ca550b-kube-api-access-4pl4b\" (UniqueName: \"kubernetes.io/projected/c603fe2e-2417-4088-945a-199a01ca550b-kube-api-access-4pl4b\") pod \"c603fe2e-2417-4088-945a-199a01ca550b\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") "
Oct 31 19:56:49.208556 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.208291    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-dshm\") pod \"c603fe2e-2417-4088-945a-199a01ca550b\" (UID: \"c603fe2e-2417-4088-945a-199a01ca550b\") "
Oct 31 19:56:49.208556 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.208369    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-home" pod "c603fe2e-2417-4088-945a-199a01ca550b" (UID: "c603fe2e-2417-4088-945a-199a01ca550b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:49.208556 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.208508    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-tmp-dir" pod "c603fe2e-2417-4088-945a-199a01ca550b" (UID: "c603fe2e-2417-4088-945a-199a01ca550b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:49.208556 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.208525    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-model-cache" pod "c603fe2e-2417-4088-945a-199a01ca550b" (UID: "c603fe2e-2417-4088-945a-199a01ca550b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:49.208766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.208674    4484 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-home\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:49.208766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.208689    4484 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-tmp-dir\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:49.208766 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.208697    4484 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-model-cache\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:49.210464 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.210436    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c603fe2e-2417-4088-945a-199a01ca550b-tls-certs" pod "c603fe2e-2417-4088-945a-199a01ca550b" (UID: "c603fe2e-2417-4088-945a-199a01ca550b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:56:49.210595 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.210544    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c603fe2e-2417-4088-945a-199a01ca550b-kube-api-access-4pl4b" pod "c603fe2e-2417-4088-945a-199a01ca550b" (UID: "c603fe2e-2417-4088-945a-199a01ca550b"). InnerVolumeSpecName "kube-api-access-4pl4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:49.210645 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.210628    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-dshm" pod "c603fe2e-2417-4088-945a-199a01ca550b" (UID: "c603fe2e-2417-4088-945a-199a01ca550b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:49.311983 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.311885    4484 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c603fe2e-2417-4088-945a-199a01ca550b-tls-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:49.311983 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.311927    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pl4b\" (UniqueName: \"kubernetes.io/projected/c603fe2e-2417-4088-945a-199a01ca550b-kube-api-access-4pl4b\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:49.311983 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.311943    4484 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c603fe2e-2417-4088-945a-199a01ca550b-dshm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:56:49.450201 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.450170    4484 generic.go:356] "Generic (PLEG): container finished" podID="c603fe2e-2417-4088-945a-199a01ca550b" containerID="b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a" exitCode=0
Oct 31 19:56:49.450375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.450207    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q" event={"ID":"c603fe2e-2417-4088-945a-199a01ca550b","Type":"ContainerDied","Data":"b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a"}
Oct 31 19:56:49.450375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.450224    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"
Oct 31 19:56:49.450375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.450233    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q" event={"ID":"c603fe2e-2417-4088-945a-199a01ca550b","Type":"ContainerDied","Data":"f6e826dfffd1ea904833a227eb9cbe3f4ed6a8d05f59fc5bbc254c840887f560"}
Oct 31 19:56:49.450375 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.450249    4484 scope.go:122] "RemoveContainer" containerID="b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a"
Oct 31 19:56:49.460073 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.460057    4484 scope.go:122] "RemoveContainer" containerID="b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a"
Oct 31 19:56:49.460348 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:56:49.460332    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a\": container with ID starting with b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a not found: ID does not exist" containerID="b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a"
Oct 31 19:56:49.460396 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.460356    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a"} err="failed to get container status \"b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a\": rpc error: code = NotFound desc = could not find container \"b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a\": container with ID starting with b52edf3280a18d889f1bf168b69ab4e7916e92066d983c0741d1ed0b21b8d24a not found: ID does not exist"
Oct 31 19:56:49.619795 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.619711    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"]
Oct 31 19:56:49.621414 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.621393    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-20b73ef4-kserve-64bb59945-8454q"]
Oct 31 19:56:49.992558 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:49.992533    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c603fe2e-2417-4088-945a-199a01ca550b" path="/var/lib/kubelet/pods/c603fe2e-2417-4088-945a-199a01ca550b/volumes"
Oct 31 19:56:50.966866 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:50.966828    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:56:55.467371 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:56:55.467334    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:56:56.273906 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:56.273860    4484 scope.go:122] "RemoveContainer" containerID="6d861385b3cbf620d8124088178b904cea71033d3357c09215f2088b88d2870d"
Oct 31 19:56:56.284192 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:56.284150    4484 scope.go:122] "RemoveContainer" containerID="958deb839eebee9798d788981dc63ebc58046b1e6064a2e5540f112ddef30488"
Oct 31 19:56:56.294033 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:56.294009    4484 scope.go:122] "RemoveContainer" containerID="08ff3502933a284dfb70f1b49c976ffa1576d0575d8e6e231ce6eab9c262b339"
Oct 31 19:56:59.446315 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:56:59.446268    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:57:01.364972 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.364929    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"]
Oct 31 19:57:01.365594 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.365575    4484 cpu_manager.go:414] "containerMap: removing container" podUID="48d92140-c003-46e2-9717-9ddc11015581" containerName="revoke-keys"
Oct 31 19:57:01.365710 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.365596    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d92140-c003-46e2-9717-9ddc11015581" containerName="revoke-keys"
Oct 31 19:57:01.365710 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.365621    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c603fe2e-2417-4088-945a-199a01ca550b" containerName="main"
Oct 31 19:57:01.365710 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.365628    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c603fe2e-2417-4088-945a-199a01ca550b" containerName="main"
Oct 31 19:57:01.365871 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.365718    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c603fe2e-2417-4088-945a-199a01ca550b" containerName="main"
Oct 31 19:57:01.365871 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.365741    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d92140-c003-46e2-9717-9ddc11015581" containerName="revoke-keys"
Oct 31 19:57:01.370275 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.370254    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:01.424890 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.424863    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"]
Oct 31 19:57:01.465307 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.465270    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lnbd\" (UniqueName: \"kubernetes.io/projected/a29d5139-b34b-44c5-803d-49ad53f1609d-kube-api-access-6lnbd\") pod \"maas-api-revoke-sub-6f166acd-wwxv4\" (UID: \"a29d5139-b34b-44c5-803d-49ad53f1609d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:01.465467 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.465418    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f166acd-wwxv4\" (UID: \"a29d5139-b34b-44c5-803d-49ad53f1609d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:01.465529 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.465494    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f166acd-wwxv4\" (UID: \"a29d5139-b34b-44c5-803d-49ad53f1609d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:01.473398 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.473375    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:57:01.473565 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.473439    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 19:57:01.566908 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.566880    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lnbd\" (UniqueName: \"kubernetes.io/projected/a29d5139-b34b-44c5-803d-49ad53f1609d-kube-api-access-6lnbd\") pod \"maas-api-revoke-sub-6f166acd-wwxv4\" (UID: \"a29d5139-b34b-44c5-803d-49ad53f1609d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:01.567081 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.566960    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f166acd-wwxv4\" (UID: \"a29d5139-b34b-44c5-803d-49ad53f1609d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:01.567081 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.566983    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f166acd-wwxv4\" (UID: \"a29d5139-b34b-44c5-803d-49ad53f1609d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:01.567663 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.567641    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f166acd-wwxv4\" (UID: \"a29d5139-b34b-44c5-803d-49ad53f1609d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:01.569514 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.569467    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f166acd-wwxv4\" (UID: \"a29d5139-b34b-44c5-803d-49ad53f1609d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:01.625780 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.625718    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lnbd\" (UniqueName: \"kubernetes.io/projected/a29d5139-b34b-44c5-803d-49ad53f1609d-kube-api-access-6lnbd\") pod \"maas-api-revoke-sub-6f166acd-wwxv4\" (UID: \"a29d5139-b34b-44c5-803d-49ad53f1609d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:01.683231 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:01.683199    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:02.051677 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:02.051635    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"]
Oct 31 19:57:02.054307 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:57:02.054276    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29d5139_b34b_44c5_803d_49ad53f1609d.slice/crio-f4873a38e568b3755d1a7f0d20987fbc3e07982b6ac1503149ef1085b9d7945e WatchSource:0}: Error finding container f4873a38e568b3755d1a7f0d20987fbc3e07982b6ac1503149ef1085b9d7945e: Status 404 returned error can't find the container with id f4873a38e568b3755d1a7f0d20987fbc3e07982b6ac1503149ef1085b9d7945e
Oct 31 19:57:02.560389 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:02.560353    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4" event={"ID":"a29d5139-b34b-44c5-803d-49ad53f1609d","Type":"ContainerStarted","Data":"27adc882ea591e1a9041449a09e14a9e5f7ce70c5cc26360c64e6f37a1d6b615"}
Oct 31 19:57:02.560821 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:02.560410    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4" event={"ID":"a29d5139-b34b-44c5-803d-49ad53f1609d","Type":"ContainerStarted","Data":"f4873a38e568b3755d1a7f0d20987fbc3e07982b6ac1503149ef1085b9d7945e"}
Oct 31 19:57:02.677297 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:02.677184    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4" podStartSLOduration=1.677169613 podStartE2EDuration="1.677169613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:02.675792805 +0000 UTC m=+5768.822242670" watchObservedRunningTime="2026-10-31 19:57:02.677169613 +0000 UTC m=+5768.823619477"
Oct 31 19:57:03.566991 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:03.566955    4484 generic.go:356] "Generic (PLEG): container finished" podID="a29d5139-b34b-44c5-803d-49ad53f1609d" containerID="27adc882ea591e1a9041449a09e14a9e5f7ce70c5cc26360c64e6f37a1d6b615" exitCode=0
Oct 31 19:57:03.567445 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:03.567042    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4" event={"ID":"a29d5139-b34b-44c5-803d-49ad53f1609d","Type":"ContainerDied","Data":"27adc882ea591e1a9041449a09e14a9e5f7ce70c5cc26360c64e6f37a1d6b615"}
Oct 31 19:57:04.712530 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:04.712504    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:04.798208 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:04.798181    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29d5139-b34b-44c5-803d-49ad53f1609d-kube-api-access-6lnbd\" (UniqueName: \"kubernetes.io/projected/a29d5139-b34b-44c5-803d-49ad53f1609d-kube-api-access-6lnbd\") pod \"a29d5139-b34b-44c5-803d-49ad53f1609d\" (UID: \"a29d5139-b34b-44c5-803d-49ad53f1609d\") "
Oct 31 19:57:04.798371 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:04.798228    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-xks-ca\") pod \"a29d5139-b34b-44c5-803d-49ad53f1609d\" (UID: \"a29d5139-b34b-44c5-803d-49ad53f1609d\") "
Oct 31 19:57:04.798371 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:04.798270    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-openshift-ca\") pod \"a29d5139-b34b-44c5-803d-49ad53f1609d\" (UID: \"a29d5139-b34b-44c5-803d-49ad53f1609d\") "
Oct 31 19:57:04.798705 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:04.798676    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-openshift-ca" pod "a29d5139-b34b-44c5-803d-49ad53f1609d" (UID: "a29d5139-b34b-44c5-803d-49ad53f1609d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:57:04.800367 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:04.800346    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-xks-ca" pod "a29d5139-b34b-44c5-803d-49ad53f1609d" (UID: "a29d5139-b34b-44c5-803d-49ad53f1609d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:57:04.800459 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:04.800365    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29d5139-b34b-44c5-803d-49ad53f1609d-kube-api-access-6lnbd" pod "a29d5139-b34b-44c5-803d-49ad53f1609d" (UID: "a29d5139-b34b-44c5-803d-49ad53f1609d"). InnerVolumeSpecName "kube-api-access-6lnbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:57:04.899176 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:04.899101    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:57:04.899176 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:04.899130    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lnbd\" (UniqueName: \"kubernetes.io/projected/a29d5139-b34b-44c5-803d-49ad53f1609d-kube-api-access-6lnbd\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:57:04.899176 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:04.899140    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29d5139-b34b-44c5-803d-49ad53f1609d-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:57:05.586668 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:05.586638    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"
Oct 31 19:57:05.586840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:05.586671    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4" event={"ID":"a29d5139-b34b-44c5-803d-49ad53f1609d","Type":"ContainerDied","Data":"f4873a38e568b3755d1a7f0d20987fbc3e07982b6ac1503149ef1085b9d7945e"}
Oct 31 19:57:05.586840 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:05.586737    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4873a38e568b3755d1a7f0d20987fbc3e07982b6ac1503149ef1085b9d7945e"
Oct 31 19:57:14.105059 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:14.105027    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:57:18.151536 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.151503    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"]
Oct 31 19:57:18.151928 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.151907    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a29d5139-b34b-44c5-803d-49ad53f1609d" containerName="revoke-keys"
Oct 31 19:57:18.151928 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.151918    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29d5139-b34b-44c5-803d-49ad53f1609d" containerName="revoke-keys"
Oct 31 19:57:18.152006 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.151979    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29d5139-b34b-44c5-803d-49ad53f1609d" containerName="revoke-keys"
Oct 31 19:57:18.155041 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.155025    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"
Oct 31 19:57:18.207740 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.207712    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-plugins-config-volume\") pod \"payload-pre-processing-9b47dd48d-78kkm\" (UID: \"3dc0ccff-4c2b-41a8-88ce-268404c1fefc\") " pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"
Oct 31 19:57:18.207885 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.207771    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87n78\" (UniqueName: \"kubernetes.io/projected/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-kube-api-access-87n78\") pod \"payload-pre-processing-9b47dd48d-78kkm\" (UID: \"3dc0ccff-4c2b-41a8-88ce-268404c1fefc\") " pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"
Oct 31 19:57:18.214494 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.214451    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"]
Oct 31 19:57:18.308541 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.308509    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-plugins-config-volume\") pod \"payload-pre-processing-9b47dd48d-78kkm\" (UID: \"3dc0ccff-4c2b-41a8-88ce-268404c1fefc\") " pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"
Oct 31 19:57:18.308732 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.308570    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87n78\" (UniqueName: \"kubernetes.io/projected/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-kube-api-access-87n78\") pod \"payload-pre-processing-9b47dd48d-78kkm\" (UID: \"3dc0ccff-4c2b-41a8-88ce-268404c1fefc\") " pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"
Oct 31 19:57:18.309161 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.309140    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-plugins-config-volume\") pod \"payload-pre-processing-9b47dd48d-78kkm\" (UID: \"3dc0ccff-4c2b-41a8-88ce-268404c1fefc\") " pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"
Oct 31 19:57:18.378410 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.378372    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87n78\" (UniqueName: \"kubernetes.io/projected/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-kube-api-access-87n78\") pod \"payload-pre-processing-9b47dd48d-78kkm\" (UID: \"3dc0ccff-4c2b-41a8-88ce-268404c1fefc\") " pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"
Oct 31 19:57:18.467655 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.467580    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"
Oct 31 19:57:18.844080 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:18.844057    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"]
Oct 31 19:57:18.845668 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:57:18.845638    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc0ccff_4c2b_41a8_88ce_268404c1fefc.slice/crio-9f9d63c7e5d56ff8200c2875ca8006d16daddefc26db38d3cc661bb67db80b1f WatchSource:0}: Error finding container 9f9d63c7e5d56ff8200c2875ca8006d16daddefc26db38d3cc661bb67db80b1f: Status 404 returned error can't find the container with id 9f9d63c7e5d56ff8200c2875ca8006d16daddefc26db38d3cc661bb67db80b1f
Oct 31 19:57:19.653755 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:19.653723    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm" event={"ID":"3dc0ccff-4c2b-41a8-88ce-268404c1fefc","Type":"ContainerStarted","Data":"25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301"}
Oct 31 19:57:19.654163 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:19.653803    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm" event={"ID":"3dc0ccff-4c2b-41a8-88ce-268404c1fefc","Type":"ContainerStarted","Data":"9f9d63c7e5d56ff8200c2875ca8006d16daddefc26db38d3cc661bb67db80b1f"}
Oct 31 19:57:19.654163 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:19.653835    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"
Oct 31 19:57:19.774165 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:19.774117    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm" podStartSLOduration=1.774105461 podStartE2EDuration="1.774105461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:19.771106481 +0000 UTC m=+5785.917556347" watchObservedRunningTime="2026-10-31 19:57:19.774105461 +0000 UTC m=+5785.920555389"
Oct 31 19:57:25.467597 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:57:25.467563    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:57:30.661381 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:30.661353    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"
Oct 31 19:57:30.830300 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:30.830264    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"]
Oct 31 19:57:32.888543 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:32.888501    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7498df5bb7-vnhf9"]
Oct 31 19:57:32.892188 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:32.892172    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9"
Oct 31 19:57:32.946403 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:32.946370    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl5ct\" (UniqueName: \"kubernetes.io/projected/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-kube-api-access-xl5ct\") pod \"payload-processing-7498df5bb7-vnhf9\" (UID: \"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf\") " pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9"
Oct 31 19:57:32.946572 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:32.946440    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-plugins-config-volume\") pod \"payload-processing-7498df5bb7-vnhf9\" (UID: \"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf\") " pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9"
Oct 31 19:57:32.951221 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:32.951195    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7498df5bb7-vnhf9"]
Oct 31 19:57:33.047612 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:33.047585    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl5ct\" (UniqueName: \"kubernetes.io/projected/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-kube-api-access-xl5ct\") pod \"payload-processing-7498df5bb7-vnhf9\" (UID: \"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf\") " pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9"
Oct 31 19:57:33.047789 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:33.047639    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-plugins-config-volume\") pod \"payload-processing-7498df5bb7-vnhf9\" (UID: \"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf\") " pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9"
Oct 31 19:57:33.048322 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:33.048296    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-plugins-config-volume\") pod \"payload-processing-7498df5bb7-vnhf9\" (UID: \"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf\") " pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9"
Oct 31 19:57:33.111547 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:33.111510    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl5ct\" (UniqueName: \"kubernetes.io/projected/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-kube-api-access-xl5ct\") pod \"payload-processing-7498df5bb7-vnhf9\" (UID: \"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf\") " pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9"
Oct 31 19:57:33.205079 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:33.205009    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9"
Oct 31 19:57:33.575487 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:33.575443    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7498df5bb7-vnhf9"]
Oct 31 19:57:33.576671 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:57:33.576643    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbd1d01_5fc0_4084_8f32_f1b22bf807cf.slice/crio-113d63d46baf9e6ab16919c724c5c5ac0186d79af1ad97c937ed71840f5db770 WatchSource:0}: Error finding container 113d63d46baf9e6ab16919c724c5c5ac0186d79af1ad97c937ed71840f5db770: Status 404 returned error can't find the container with id 113d63d46baf9e6ab16919c724c5c5ac0186d79af1ad97c937ed71840f5db770
Oct 31 19:57:33.720096 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:33.720067    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9" event={"ID":"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf","Type":"ContainerStarted","Data":"a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933"}
Oct 31 19:57:33.720268 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:33.720114    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9" event={"ID":"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf","Type":"ContainerStarted","Data":"113d63d46baf9e6ab16919c724c5c5ac0186d79af1ad97c937ed71840f5db770"}
Oct 31 19:57:33.720268 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:33.720187    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9"
Oct 31 19:57:33.838681 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:33.838582    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9" podStartSLOduration=1.8385681059999999 podStartE2EDuration="1.838568106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:33.837768586 +0000 UTC m=+5799.984218455" watchObservedRunningTime="2026-10-31 19:57:33.838568106 +0000 UTC m=+5799.985017970"
Oct 31 19:57:35.830747 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:35.830656    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm" podUID="93867b3f-070c-4791-b3a9-55431f5cd3a4" containerName="payload-pre-processing" containerID="cri-o://8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a" gracePeriod=25
Oct 31 19:57:36.092765 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.092704    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"
Oct 31 19:57:36.176113 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.176081    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93867b3f-070c-4791-b3a9-55431f5cd3a4-kube-api-access-kj627\" (UniqueName: \"kubernetes.io/projected/93867b3f-070c-4791-b3a9-55431f5cd3a4-kube-api-access-kj627\") pod \"93867b3f-070c-4791-b3a9-55431f5cd3a4\" (UID: \"93867b3f-070c-4791-b3a9-55431f5cd3a4\") "
Oct 31 19:57:36.176309 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.176128    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93867b3f-070c-4791-b3a9-55431f5cd3a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93867b3f-070c-4791-b3a9-55431f5cd3a4-plugins-config-volume\") pod \"93867b3f-070c-4791-b3a9-55431f5cd3a4\" (UID: \"93867b3f-070c-4791-b3a9-55431f5cd3a4\") "
Oct 31 19:57:36.176573 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.176550    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93867b3f-070c-4791-b3a9-55431f5cd3a4-plugins-config-volume" pod "93867b3f-070c-4791-b3a9-55431f5cd3a4" (UID: "93867b3f-070c-4791-b3a9-55431f5cd3a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:57:36.178364 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.178338    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93867b3f-070c-4791-b3a9-55431f5cd3a4-kube-api-access-kj627" pod "93867b3f-070c-4791-b3a9-55431f5cd3a4" (UID: "93867b3f-070c-4791-b3a9-55431f5cd3a4"). InnerVolumeSpecName "kube-api-access-kj627". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:57:36.277559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.277523    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj627\" (UniqueName: \"kubernetes.io/projected/93867b3f-070c-4791-b3a9-55431f5cd3a4-kube-api-access-kj627\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:57:36.277559 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.277556    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93867b3f-070c-4791-b3a9-55431f5cd3a4-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:57:36.735811 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.735774    4484 generic.go:356] "Generic (PLEG): container finished" podID="93867b3f-070c-4791-b3a9-55431f5cd3a4" containerID="8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a" exitCode=0
Oct 31 19:57:36.735987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.735829    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"
Oct 31 19:57:36.735987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.735869    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm" event={"ID":"93867b3f-070c-4791-b3a9-55431f5cd3a4","Type":"ContainerDied","Data":"8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a"}
Oct 31 19:57:36.735987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.735934    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm" event={"ID":"93867b3f-070c-4791-b3a9-55431f5cd3a4","Type":"ContainerDied","Data":"859a8e38c37cf28b83298ea9e80d27321619c54506b8e35152de232f5f9cca0c"}
Oct 31 19:57:36.735987 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.735964    4484 scope.go:122] "RemoveContainer" containerID="8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a"
Oct 31 19:57:36.745259 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.745236    4484 scope.go:122] "RemoveContainer" containerID="8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a"
Oct 31 19:57:36.745554 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:57:36.745536    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a\": container with ID starting with 8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a not found: ID does not exist" containerID="8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a"
Oct 31 19:57:36.745615 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.745562    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a"} err="failed to get container status \"8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a\": rpc error: code = NotFound desc = could not find container \"8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a\": container with ID starting with 8a36efdf43843e06423f37b407497f13ec6c5b00c60e1e14ef51f745ce3b613a not found: ID does not exist"
Oct 31 19:57:36.903136 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.903104    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"]
Oct 31 19:57:36.954406 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:36.954374    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6ksnm"]
Oct 31 19:57:37.992131 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:37.992098    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93867b3f-070c-4791-b3a9-55431f5cd3a4" path="/var/lib/kubelet/pods/93867b3f-070c-4791-b3a9-55431f5cd3a4/volumes"
Oct 31 19:57:44.728883 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:44.728853    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9"
Oct 31 19:57:44.898222 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:44.898188    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-62pcj"]
Oct 31 19:57:47.090672 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.090638    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-596fc796b7-8thbk"]
Oct 31 19:57:47.091056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.091036    4484 cpu_manager.go:414] "containerMap: removing container" podUID="93867b3f-070c-4791-b3a9-55431f5cd3a4" containerName="payload-pre-processing"
Oct 31 19:57:47.091056 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.091046    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93867b3f-070c-4791-b3a9-55431f5cd3a4" containerName="payload-pre-processing"
Oct 31 19:57:47.091130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.091121    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="93867b3f-070c-4791-b3a9-55431f5cd3a4" containerName="payload-pre-processing"
Oct 31 19:57:47.094105 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.094090    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-596fc796b7-8thbk"
Oct 31 19:57:47.153122 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.153087    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-596fc796b7-8thbk"]
Oct 31 19:57:47.178036 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.178006    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b48a0e7-657c-46cf-9818-6bb3428fc94c-plugins-config-volume\") pod \"payload-processing-596fc796b7-8thbk\" (UID: \"0b48a0e7-657c-46cf-9818-6bb3428fc94c\") " pod="openshift-ingress/payload-processing-596fc796b7-8thbk"
Oct 31 19:57:47.178198 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.178056    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbnjn\" (UniqueName: \"kubernetes.io/projected/0b48a0e7-657c-46cf-9818-6bb3428fc94c-kube-api-access-wbnjn\") pod \"payload-processing-596fc796b7-8thbk\" (UID: \"0b48a0e7-657c-46cf-9818-6bb3428fc94c\") " pod="openshift-ingress/payload-processing-596fc796b7-8thbk"
Oct 31 19:57:47.257502 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.257456    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"]
Oct 31 19:57:47.261071 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.261055    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"
Oct 31 19:57:47.278816 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.278791    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b48a0e7-657c-46cf-9818-6bb3428fc94c-plugins-config-volume\") pod \"payload-processing-596fc796b7-8thbk\" (UID: \"0b48a0e7-657c-46cf-9818-6bb3428fc94c\") " pod="openshift-ingress/payload-processing-596fc796b7-8thbk"
Oct 31 19:57:47.278951 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.278835    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbnjn\" (UniqueName: \"kubernetes.io/projected/0b48a0e7-657c-46cf-9818-6bb3428fc94c-kube-api-access-wbnjn\") pod \"payload-processing-596fc796b7-8thbk\" (UID: \"0b48a0e7-657c-46cf-9818-6bb3428fc94c\") " pod="openshift-ingress/payload-processing-596fc796b7-8thbk"
Oct 31 19:57:47.279538 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.279518    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b48a0e7-657c-46cf-9818-6bb3428fc94c-plugins-config-volume\") pod \"payload-processing-596fc796b7-8thbk\" (UID: \"0b48a0e7-657c-46cf-9818-6bb3428fc94c\") " pod="openshift-ingress/payload-processing-596fc796b7-8thbk"
Oct 31 19:57:47.318618 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.318592    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"]
Oct 31 19:57:47.338677 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.338645    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbnjn\" (UniqueName: \"kubernetes.io/projected/0b48a0e7-657c-46cf-9818-6bb3428fc94c-kube-api-access-wbnjn\") pod \"payload-processing-596fc796b7-8thbk\" (UID: \"0b48a0e7-657c-46cf-9818-6bb3428fc94c\") " pod="openshift-ingress/payload-processing-596fc796b7-8thbk"
Oct 31 19:57:47.379650 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.379587    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7f6m\" (UniqueName: \"kubernetes.io/projected/7856c13c-9c85-4d24-9149-55953e517e57-kube-api-access-f7f6m\") pod \"payload-pre-processing-86cf587c44-nbkn8\" (UID: \"7856c13c-9c85-4d24-9149-55953e517e57\") " pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"
Oct 31 19:57:47.379650 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.379627    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7856c13c-9c85-4d24-9149-55953e517e57-plugins-config-volume\") pod \"payload-pre-processing-86cf587c44-nbkn8\" (UID: \"7856c13c-9c85-4d24-9149-55953e517e57\") " pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"
Oct 31 19:57:47.406648 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.406611    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-596fc796b7-8thbk"
Oct 31 19:57:47.480398 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.480351    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7f6m\" (UniqueName: \"kubernetes.io/projected/7856c13c-9c85-4d24-9149-55953e517e57-kube-api-access-f7f6m\") pod \"payload-pre-processing-86cf587c44-nbkn8\" (UID: \"7856c13c-9c85-4d24-9149-55953e517e57\") " pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"
Oct 31 19:57:47.480591 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.480422    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7856c13c-9c85-4d24-9149-55953e517e57-plugins-config-volume\") pod \"payload-pre-processing-86cf587c44-nbkn8\" (UID: \"7856c13c-9c85-4d24-9149-55953e517e57\") " pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"
Oct 31 19:57:47.481327 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.481303    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7856c13c-9c85-4d24-9149-55953e517e57-plugins-config-volume\") pod \"payload-pre-processing-86cf587c44-nbkn8\" (UID: \"7856c13c-9c85-4d24-9149-55953e517e57\") " pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"
Oct 31 19:57:47.539065 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.539029    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7f6m\" (UniqueName: \"kubernetes.io/projected/7856c13c-9c85-4d24-9149-55953e517e57-kube-api-access-f7f6m\") pod \"payload-pre-processing-86cf587c44-nbkn8\" (UID: \"7856c13c-9c85-4d24-9149-55953e517e57\") " pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"
Oct 31 19:57:47.580634 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.580594    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"
Oct 31 19:57:47.782060 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.782029    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-596fc796b7-8thbk"]
Oct 31 19:57:47.784261 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:57:47.784233    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b48a0e7_657c_46cf_9818_6bb3428fc94c.slice/crio-0a170cd8e7ae1a8ef9e9279a8839d61b6356785a49ff4ac0db68688ceb06fed7 WatchSource:0}: Error finding container 0a170cd8e7ae1a8ef9e9279a8839d61b6356785a49ff4ac0db68688ceb06fed7: Status 404 returned error can't find the container with id 0a170cd8e7ae1a8ef9e9279a8839d61b6356785a49ff4ac0db68688ceb06fed7
Oct 31 19:57:47.796853 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.796763    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-596fc796b7-8thbk" event={"ID":"0b48a0e7-657c-46cf-9818-6bb3428fc94c","Type":"ContainerStarted","Data":"0a170cd8e7ae1a8ef9e9279a8839d61b6356785a49ff4ac0db68688ceb06fed7"}
Oct 31 19:57:47.955382 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:47.955354    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"]
Oct 31 19:57:47.956678 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:57:47.956651    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7856c13c_9c85_4d24_9149_55953e517e57.slice/crio-a267377eb068ae59a8fe0203147fbc28039d7ec36914b2f4501f703e09d57d9c WatchSource:0}: Error finding container a267377eb068ae59a8fe0203147fbc28039d7ec36914b2f4501f703e09d57d9c: Status 404 returned error can't find the container with id a267377eb068ae59a8fe0203147fbc28039d7ec36914b2f4501f703e09d57d9c
Oct 31 19:57:48.803699 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:48.803659    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-596fc796b7-8thbk" event={"ID":"0b48a0e7-657c-46cf-9818-6bb3428fc94c","Type":"ContainerStarted","Data":"0880a4ebef0a46a64103d3a57b36dac7205c3af9400e7b1c8d3da5b91381bde2"}
Oct 31 19:57:48.804176 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:48.803745    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-596fc796b7-8thbk"
Oct 31 19:57:48.805169 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:48.805147    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8" event={"ID":"7856c13c-9c85-4d24-9149-55953e517e57","Type":"ContainerStarted","Data":"c19f7e8d0f0c2b4cd92ff6a4e251b36d0ac0100e78652555124962f54fc16d7a"}
Oct 31 19:57:48.805279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:48.805181    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8" event={"ID":"7856c13c-9c85-4d24-9149-55953e517e57","Type":"ContainerStarted","Data":"a267377eb068ae59a8fe0203147fbc28039d7ec36914b2f4501f703e09d57d9c"}
Oct 31 19:57:48.805279 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:48.805202    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"
Oct 31 19:57:48.924352 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:48.924304    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-596fc796b7-8thbk" podStartSLOduration=1.924291529 podStartE2EDuration="1.924291529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:48.922080457 +0000 UTC m=+5815.068530320" watchObservedRunningTime="2026-10-31 19:57:48.924291529 +0000 UTC m=+5815.070741391"
Oct 31 19:57:49.038631 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:49.038579    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8" podStartSLOduration=2.038567029 podStartE2EDuration="2.038567029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:49.037977138 +0000 UTC m=+5815.184427019" watchObservedRunningTime="2026-10-31 19:57:49.038567029 +0000 UTC m=+5815.185016892"
Oct 31 19:57:49.898745 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:49.898700    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj" podUID="a44a1311-1de6-46c3-8070-07fa09d89854" containerName="payload-processing" containerID="cri-o://3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f" gracePeriod=25
Oct 31 19:57:50.154389 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.154331    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj"
Oct 31 19:57:50.310092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.310061    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44a1311-1de6-46c3-8070-07fa09d89854-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44a1311-1de6-46c3-8070-07fa09d89854-plugins-config-volume\") pod \"a44a1311-1de6-46c3-8070-07fa09d89854\" (UID: \"a44a1311-1de6-46c3-8070-07fa09d89854\") "
Oct 31 19:57:50.310296 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.310117    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44a1311-1de6-46c3-8070-07fa09d89854-kube-api-access-llqgr\" (UniqueName: \"kubernetes.io/projected/a44a1311-1de6-46c3-8070-07fa09d89854-kube-api-access-llqgr\") pod \"a44a1311-1de6-46c3-8070-07fa09d89854\" (UID: \"a44a1311-1de6-46c3-8070-07fa09d89854\") "
Oct 31 19:57:50.310715 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.310685    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44a1311-1de6-46c3-8070-07fa09d89854-plugins-config-volume" pod "a44a1311-1de6-46c3-8070-07fa09d89854" (UID: "a44a1311-1de6-46c3-8070-07fa09d89854"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:57:50.312293 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.312269    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44a1311-1de6-46c3-8070-07fa09d89854-kube-api-access-llqgr" pod "a44a1311-1de6-46c3-8070-07fa09d89854" (UID: "a44a1311-1de6-46c3-8070-07fa09d89854"). InnerVolumeSpecName "kube-api-access-llqgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:57:50.411660 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.411576    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llqgr\" (UniqueName: \"kubernetes.io/projected/a44a1311-1de6-46c3-8070-07fa09d89854-kube-api-access-llqgr\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:57:50.411660 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.411607    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44a1311-1de6-46c3-8070-07fa09d89854-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:57:50.816547 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.816515    4484 generic.go:356] "Generic (PLEG): container finished" podID="a44a1311-1de6-46c3-8070-07fa09d89854" containerID="3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f" exitCode=0
Oct 31 19:57:50.816735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.816566    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj"
Oct 31 19:57:50.816735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.816605    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj" event={"ID":"a44a1311-1de6-46c3-8070-07fa09d89854","Type":"ContainerDied","Data":"3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f"}
Oct 31 19:57:50.816735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.816672    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-62pcj" event={"ID":"a44a1311-1de6-46c3-8070-07fa09d89854","Type":"ContainerDied","Data":"228487c8bf954bad71d099d766f28b4031c0b9f3b74c8b8a0f52d032f8f6533d"}
Oct 31 19:57:50.816735 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.816700    4484 scope.go:122] "RemoveContainer" containerID="3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f"
Oct 31 19:57:50.825937 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.825916    4484 scope.go:122] "RemoveContainer" containerID="3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f"
Oct 31 19:57:50.826194 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:57:50.826174    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f\": container with ID starting with 3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f not found: ID does not exist" containerID="3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f"
Oct 31 19:57:50.826268 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.826206    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f"} err="failed to get container status \"3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f\": rpc error: code = NotFound desc = could not find container \"3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f\": container with ID starting with 3894e3a6ac7133a73f369bfd4105ecd10f3f6d53a07f214389c19fa6d299300f not found: ID does not exist"
Oct 31 19:57:50.983815 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.983781    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-62pcj"]
Oct 31 19:57:50.987214 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:50.987190    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-62pcj"]
Oct 31 19:57:51.992602 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:51.992571    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44a1311-1de6-46c3-8070-07fa09d89854" path="/var/lib/kubelet/pods/a44a1311-1de6-46c3-8070-07fa09d89854/volumes"
Oct 31 19:57:55.467437 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:57:55.467401    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:57:59.813249 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:59.813218    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-86cf587c44-nbkn8"
Oct 31 19:57:59.813686 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:59.813378    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-596fc796b7-8thbk"
Oct 31 19:57:59.986694 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:57:59.986657    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"]
Oct 31 19:58:00.072891 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:00.072778    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7498df5bb7-vnhf9"]
Oct 31 19:58:04.987183 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:04.987152    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm" podUID="3dc0ccff-4c2b-41a8-88ce-268404c1fefc" containerName="payload-pre-processing" containerID="cri-o://25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301" gracePeriod=25
Oct 31 19:58:05.074142 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.074105    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9" podUID="1cbd1d01-5fc0-4084-8f32-f1b22bf807cf" containerName="payload-processing" containerID="cri-o://a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933" gracePeriod=25
Oct 31 19:58:05.348332 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.348309    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"
Oct 31 19:58:05.351130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.351111    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9"
Oct 31 19:58:05.439162 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.439136    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-kube-api-access-87n78\" (UniqueName: \"kubernetes.io/projected/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-kube-api-access-87n78\") pod \"3dc0ccff-4c2b-41a8-88ce-268404c1fefc\" (UID: \"3dc0ccff-4c2b-41a8-88ce-268404c1fefc\") "
Oct 31 19:58:05.439347 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.439231    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-kube-api-access-xl5ct\" (UniqueName: \"kubernetes.io/projected/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-kube-api-access-xl5ct\") pod \"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf\" (UID: \"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf\") "
Oct 31 19:58:05.439347 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.439283    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-plugins-config-volume\") pod \"3dc0ccff-4c2b-41a8-88ce-268404c1fefc\" (UID: \"3dc0ccff-4c2b-41a8-88ce-268404c1fefc\") "
Oct 31 19:58:05.439347 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.439327    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-plugins-config-volume\") pod \"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf\" (UID: \"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf\") "
Oct 31 19:58:05.439744 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.439714    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-plugins-config-volume" pod "3dc0ccff-4c2b-41a8-88ce-268404c1fefc" (UID: "3dc0ccff-4c2b-41a8-88ce-268404c1fefc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:58:05.440011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.439751    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-plugins-config-volume" pod "1cbd1d01-5fc0-4084-8f32-f1b22bf807cf" (UID: "1cbd1d01-5fc0-4084-8f32-f1b22bf807cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:58:05.441503 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.441467    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-kube-api-access-87n78" pod "3dc0ccff-4c2b-41a8-88ce-268404c1fefc" (UID: "3dc0ccff-4c2b-41a8-88ce-268404c1fefc"). InnerVolumeSpecName "kube-api-access-87n78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:58:05.441592 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.441515    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-kube-api-access-xl5ct" pod "1cbd1d01-5fc0-4084-8f32-f1b22bf807cf" (UID: "1cbd1d01-5fc0-4084-8f32-f1b22bf807cf"). InnerVolumeSpecName "kube-api-access-xl5ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:58:05.540916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.540837    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl5ct\" (UniqueName: \"kubernetes.io/projected/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-kube-api-access-xl5ct\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:58:05.540916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.540864    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:58:05.540916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.540874    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:58:05.540916 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.540883    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87n78\" (UniqueName: \"kubernetes.io/projected/3dc0ccff-4c2b-41a8-88ce-268404c1fefc-kube-api-access-87n78\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 19:58:05.892020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.891931    4484 generic.go:356] "Generic (PLEG): container finished" podID="1cbd1d01-5fc0-4084-8f32-f1b22bf807cf" containerID="a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933" exitCode=0
Oct 31 19:58:05.892020 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.891982    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9"
Oct 31 19:58:05.892272 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.892017    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9" event={"ID":"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf","Type":"ContainerDied","Data":"a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933"}
Oct 31 19:58:05.892272 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.892068    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7498df5bb7-vnhf9" event={"ID":"1cbd1d01-5fc0-4084-8f32-f1b22bf807cf","Type":"ContainerDied","Data":"113d63d46baf9e6ab16919c724c5c5ac0186d79af1ad97c937ed71840f5db770"}
Oct 31 19:58:05.892272 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.892092    4484 scope.go:122] "RemoveContainer" containerID="a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933"
Oct 31 19:58:05.902348 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.902320    4484 generic.go:356] "Generic (PLEG): container finished" podID="3dc0ccff-4c2b-41a8-88ce-268404c1fefc" containerID="25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301" exitCode=0
Oct 31 19:58:05.902520 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.902424    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm" event={"ID":"3dc0ccff-4c2b-41a8-88ce-268404c1fefc","Type":"ContainerDied","Data":"25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301"}
Oct 31 19:58:05.902520 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.902417    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"
Oct 31 19:58:05.902661 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.902528    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-9b47dd48d-78kkm" event={"ID":"3dc0ccff-4c2b-41a8-88ce-268404c1fefc","Type":"ContainerDied","Data":"9f9d63c7e5d56ff8200c2875ca8006d16daddefc26db38d3cc661bb67db80b1f"}
Oct 31 19:58:05.910125 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.910103    4484 scope.go:122] "RemoveContainer" containerID="a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933"
Oct 31 19:58:05.910457 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:58:05.910437    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933\": container with ID starting with a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933 not found: ID does not exist" containerID="a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933"
Oct 31 19:58:05.910562 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.910469    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933"} err="failed to get container status \"a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933\": rpc error: code = NotFound desc = could not find container \"a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933\": container with ID starting with a8ca5534a68e98ccc9259c5e4d51316b13505145dccd2ac01d6add4844fd6933 not found: ID does not exist"
Oct 31 19:58:05.910562 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.910508    4484 scope.go:122] "RemoveContainer" containerID="25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301"
Oct 31 19:58:05.918462 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.918445    4484 scope.go:122] "RemoveContainer" containerID="25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301"
Oct 31 19:58:05.918734 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:58:05.918710    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301\": container with ID starting with 25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301 not found: ID does not exist" containerID="25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301"
Oct 31 19:58:05.918794 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:05.918734    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301"} err="failed to get container status \"25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301\": rpc error: code = NotFound desc = could not find container \"25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301\": container with ID starting with 25f6318bf1d13a9dce3fd1d153b1945d6abb50bbb4160c2891386e1de307e301 not found: ID does not exist"
Oct 31 19:58:06.062935 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.062893    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7498df5bb7-vnhf9"]
Oct 31 19:58:06.113387 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.113351    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-7498df5bb7-vnhf9"]
Oct 31 19:58:06.202666 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.202580    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"]
Oct 31 19:58:06.222423 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.222384    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-9b47dd48d-78kkm"]
Oct 31 19:58:06.543232 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.543199    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"]
Oct 31 19:58:06.543620 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.543606    4484 cpu_manager.go:414] "containerMap: removing container" podUID="3dc0ccff-4c2b-41a8-88ce-268404c1fefc" containerName="payload-pre-processing"
Oct 31 19:58:06.543680 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.543621    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc0ccff-4c2b-41a8-88ce-268404c1fefc" containerName="payload-pre-processing"
Oct 31 19:58:06.543680 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.543635    4484 cpu_manager.go:414] "containerMap: removing container" podUID="1cbd1d01-5fc0-4084-8f32-f1b22bf807cf" containerName="payload-processing"
Oct 31 19:58:06.543680 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.543640    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cbd1d01-5fc0-4084-8f32-f1b22bf807cf" containerName="payload-processing"
Oct 31 19:58:06.543680 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.543671    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a44a1311-1de6-46c3-8070-07fa09d89854" containerName="payload-processing"
Oct 31 19:58:06.543680 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.543676    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44a1311-1de6-46c3-8070-07fa09d89854" containerName="payload-processing"
Oct 31 19:58:06.543831 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.543734    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44a1311-1de6-46c3-8070-07fa09d89854" containerName="payload-processing"
Oct 31 19:58:06.543831 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.543744    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc0ccff-4c2b-41a8-88ce-268404c1fefc" containerName="payload-pre-processing"
Oct 31 19:58:06.543831 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.543751    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cbd1d01-5fc0-4084-8f32-f1b22bf807cf" containerName="payload-processing"
Oct 31 19:58:06.548130 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.548114    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.605092 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.605066    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"]
Oct 31 19:58:06.605292 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.605264    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 19:58:06.605449 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.605397    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 19:58:06.649817 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.649788    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f9ql\" (UniqueName: \"kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-kube-api-access-9f9ql\") pod \"payload-processing-external-model-5bcd7997-bp5d6\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.649968 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.649832    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-bp5d6\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.649968 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.649914    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-bp5d6\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.650044 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.649965    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-bp5d6\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.652811 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.652792    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Oct 31 19:58:06.652869 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.652842    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 19:58:06.652869 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.652853    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-wv2g6\""
Oct 31 19:58:06.652869 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.652849    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 19:58:06.751081 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.751047    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f9ql\" (UniqueName: \"kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-kube-api-access-9f9ql\") pod \"payload-processing-external-model-5bcd7997-bp5d6\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.751248 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.751110    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-bp5d6\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.751248 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.751166    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-bp5d6\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.751326 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.751286    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-bp5d6\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.751820 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.751803    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-bp5d6\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.752011 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.751995    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-bp5d6\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.753919 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.753896    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-bp5d6\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.810055 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.809996    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f9ql\" (UniqueName: \"kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-kube-api-access-9f9ql\") pod \"payload-processing-external-model-5bcd7997-bp5d6\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:06.860673 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:06.860646    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:07.240195 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:07.240170    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"]
Oct 31 19:58:07.242275 ip-10-0-142-120 kubenswrapper[4484]: W1031 19:58:07.242252    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15bc65fb_d639_42b4_9a81_55527c2b6943.slice/crio-6dbb84ec4ac6ed556c38be7d2feda15ae3ddde6f15a52a3afdceb072a35cff81 WatchSource:0}: Error finding container 6dbb84ec4ac6ed556c38be7d2feda15ae3ddde6f15a52a3afdceb072a35cff81: Status 404 returned error can't find the container with id 6dbb84ec4ac6ed556c38be7d2feda15ae3ddde6f15a52a3afdceb072a35cff81
Oct 31 19:58:07.685139 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:07.685052    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 19:58:07.918111 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:07.918078    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6" event={"ID":"15bc65fb-d639-42b4-9a81-55527c2b6943","Type":"ContainerStarted","Data":"b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0"}
Oct 31 19:58:07.918284 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:07.918128    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6" event={"ID":"15bc65fb-d639-42b4-9a81-55527c2b6943","Type":"ContainerStarted","Data":"6dbb84ec4ac6ed556c38be7d2feda15ae3ddde6f15a52a3afdceb072a35cff81"}
Oct 31 19:58:07.918284 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:07.918156    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:07.994423 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:07.994393    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cbd1d01-5fc0-4084-8f32-f1b22bf807cf" path="/var/lib/kubelet/pods/1cbd1d01-5fc0-4084-8f32-f1b22bf807cf/volumes"
Oct 31 19:58:07.994946 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:07.994926    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc0ccff-4c2b-41a8-88ce-268404c1fefc" path="/var/lib/kubelet/pods/3dc0ccff-4c2b-41a8-88ce-268404c1fefc/volumes"
Oct 31 19:58:08.038384 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:08.038319    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6" podStartSLOduration=2.038300799 podStartE2EDuration="2.038300799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:08.037810796 +0000 UTC m=+5834.184260658" watchObservedRunningTime="2026-10-31 19:58:08.038300799 +0000 UTC m=+5834.184750663"
Oct 31 19:58:18.933684 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:18.933654    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 19:58:25.467231 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:58:25.467195    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:58:41.141692 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:41.141663    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"]
Oct 31 19:58:41.144336 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:41.144313    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e41bc33d-n59jj"]
Oct 31 19:58:41.991839 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:58:41.991808    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c135cd02-fa93-480d-9096-4ca318a756ba" path="/var/lib/kubelet/pods/c135cd02-fa93-480d-9096-4ca318a756ba/volumes"
Oct 31 19:58:55.468698 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:58:55.468660    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:59:25.467531 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:59:25.467496    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:59:55.467302 ip-10-0-142-120 kubenswrapper[4484]: E1031 19:59:55.467265    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 19:59:56.365195 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:59:56.365163    4484 scope.go:122] "RemoveContainer" containerID="2cccf62b95014415267f9526ee68199e2e5eda467becba84ac4a6624c37571b2"
Oct 31 19:59:56.904623 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:59:56.904591    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"]
Oct 31 19:59:59.992845 ip-10-0-142-120 kubenswrapper[4484]: I1031 19:59:59.992817    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:00:00.168239 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:00.168207    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"]
Oct 31 20:00:00.173559 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:00.173534    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"
Oct 31 20:00:00.229918 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:00.229885    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"]
Oct 31 20:00:00.273361 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:00.273280    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rk9\" (UniqueName: \"kubernetes.io/projected/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-kube-api-access-75rk9\") pod \"maas-api-key-cleanup-29891280-dcpnx\" (UID: \"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"
Oct 31 20:00:00.273361 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:00.273328    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891280-dcpnx\" (UID: \"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"
Oct 31 20:00:00.277659 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:00.277637    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:00:00.374826 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:00.374782    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rk9\" (UniqueName: \"kubernetes.io/projected/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-kube-api-access-75rk9\") pod \"maas-api-key-cleanup-29891280-dcpnx\" (UID: \"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"
Oct 31 20:00:00.375158 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:00.375135    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891280-dcpnx\" (UID: \"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"
Oct 31 20:00:00.376011 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:00.375979    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891280-dcpnx\" (UID: \"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"
Oct 31 20:00:00.434627 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:00.434559    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rk9\" (UniqueName: \"kubernetes.io/projected/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-kube-api-access-75rk9\") pod \"maas-api-key-cleanup-29891280-dcpnx\" (UID: \"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"
Oct 31 20:00:00.489905 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:00.489874    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"
Oct 31 20:00:00.994552 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:00.972767    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"]
Oct 31 20:00:01.021864 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:00:01.021831    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d97cb1_0e6f_4149_b97c_d60d8be2e6e6.slice/crio-1a93cc1f80c7844878ae9109a0d622f7eef533c848280947f4bbccde576cecd0 WatchSource:0}: Error finding container 1a93cc1f80c7844878ae9109a0d622f7eef533c848280947f4bbccde576cecd0: Status 404 returned error can't find the container with id 1a93cc1f80c7844878ae9109a0d622f7eef533c848280947f4bbccde576cecd0
Oct 31 20:00:01.503718 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:01.503680    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx" event={"ID":"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6","Type":"ContainerStarted","Data":"1a93cc1f80c7844878ae9109a0d622f7eef533c848280947f4bbccde576cecd0"}
Oct 31 20:00:01.906227 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:01.906175    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6" podUID="15bc65fb-d639-42b4-9a81-55527c2b6943" containerName="payload-processing" containerID="cri-o://b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0" gracePeriod=25
Oct 31 20:00:02.156888 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.156836    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 20:00:02.250609 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.250580    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-provider-credentials\" (UniqueName: \"kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-provider-credentials\") pod \"15bc65fb-d639-42b4-9a81-55527c2b6943\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") "
Oct 31 20:00:02.250782 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.250617    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-kube-api-access-9f9ql\" (UniqueName: \"kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-kube-api-access-9f9ql\") pod \"15bc65fb-d639-42b4-9a81-55527c2b6943\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") "
Oct 31 20:00:02.250782 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.250641    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-routing-overlay-volume\") pod \"15bc65fb-d639-42b4-9a81-55527c2b6943\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") "
Oct 31 20:00:02.250782 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.250677    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-plugins-config-volume\") pod \"15bc65fb-d639-42b4-9a81-55527c2b6943\" (UID: \"15bc65fb-d639-42b4-9a81-55527c2b6943\") "
Oct 31 20:00:02.251029 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.251007    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-routing-overlay-volume" pod "15bc65fb-d639-42b4-9a81-55527c2b6943" (UID: "15bc65fb-d639-42b4-9a81-55527c2b6943"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:02.251091 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.251074    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-plugins-config-volume" pod "15bc65fb-d639-42b4-9a81-55527c2b6943" (UID: "15bc65fb-d639-42b4-9a81-55527c2b6943"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:02.252908 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.252882    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-provider-credentials" pod "15bc65fb-d639-42b4-9a81-55527c2b6943" (UID: "15bc65fb-d639-42b4-9a81-55527c2b6943"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:02.253019 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.252966    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-kube-api-access-9f9ql" pod "15bc65fb-d639-42b4-9a81-55527c2b6943" (UID: "15bc65fb-d639-42b4-9a81-55527c2b6943"). InnerVolumeSpecName "kube-api-access-9f9ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:02.351639 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.351606    4484 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-provider-credentials\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:02.351639 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.351631    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f9ql\" (UniqueName: \"kubernetes.io/projected/15bc65fb-d639-42b4-9a81-55527c2b6943-kube-api-access-9f9ql\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:02.351639 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.351640    4484 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-routing-overlay-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:02.351873 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.351649    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bc65fb-d639-42b4-9a81-55527c2b6943-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:02.509529 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.509495    4484 generic.go:356] "Generic (PLEG): container finished" podID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" containerID="18477e2f8463daa9502213f5155e89d93621f37f485b50eb9d984e2c78cf65e8" exitCode=60
Oct 31 20:00:02.509715 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.509585    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx" event={"ID":"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6","Type":"ContainerDied","Data":"18477e2f8463daa9502213f5155e89d93621f37f485b50eb9d984e2c78cf65e8"}
Oct 31 20:00:02.509851 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.509837    4484 scope.go:122] "RemoveContainer" containerID="18477e2f8463daa9502213f5155e89d93621f37f485b50eb9d984e2c78cf65e8"
Oct 31 20:00:02.510956 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.510936    4484 generic.go:356] "Generic (PLEG): container finished" podID="15bc65fb-d639-42b4-9a81-55527c2b6943" containerID="b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0" exitCode=0
Oct 31 20:00:02.511065 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.511005    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"
Oct 31 20:00:02.511065 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.511012    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6" event={"ID":"15bc65fb-d639-42b4-9a81-55527c2b6943","Type":"ContainerDied","Data":"b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0"}
Oct 31 20:00:02.511065 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.511043    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6" event={"ID":"15bc65fb-d639-42b4-9a81-55527c2b6943","Type":"ContainerDied","Data":"6dbb84ec4ac6ed556c38be7d2feda15ae3ddde6f15a52a3afdceb072a35cff81"}
Oct 31 20:00:02.511065 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.511063    4484 scope.go:122] "RemoveContainer" containerID="b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0"
Oct 31 20:00:02.522022 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.521998    4484 scope.go:122] "RemoveContainer" containerID="b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0"
Oct 31 20:00:02.522366 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:00:02.522287    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0\": container with ID starting with b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0 not found: ID does not exist" containerID="b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0"
Oct 31 20:00:02.522366 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.522313    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0"} err="failed to get container status \"b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0\": rpc error: code = NotFound desc = could not find container \"b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0\": container with ID starting with b35cc192b44adf064a4e130ebc9d7ff59609abb5b9eeae128059f1418a4c83b0 not found: ID does not exist"
Oct 31 20:00:02.739590 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.739559    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"]
Oct 31 20:00:02.742968 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:02.742937    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-bp5d6"]
Oct 31 20:00:03.525548 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:03.525503    4484 generic.go:356] "Generic (PLEG): container finished" podID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" containerID="aee28cf83cab450dc105fb0eb001dd262db219fd7b9e6aa59e4318eddd580359" exitCode=60
Oct 31 20:00:03.526068 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:03.525574    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx" event={"ID":"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6","Type":"ContainerDied","Data":"aee28cf83cab450dc105fb0eb001dd262db219fd7b9e6aa59e4318eddd580359"}
Oct 31 20:00:03.526068 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:03.525676    4484 scope.go:122] "RemoveContainer" containerID="18477e2f8463daa9502213f5155e89d93621f37f485b50eb9d984e2c78cf65e8"
Oct 31 20:00:03.526068 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:03.525942    4484 scope.go:122] "RemoveContainer" containerID="aee28cf83cab450dc105fb0eb001dd262db219fd7b9e6aa59e4318eddd580359"
Oct 31 20:00:03.526263 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:00:03.526190    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891280-dcpnx_odh-ai-gateway-infra(c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6"
Oct 31 20:00:03.992308 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:03.992275    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15bc65fb-d639-42b4-9a81-55527c2b6943" path="/var/lib/kubelet/pods/15bc65fb-d639-42b4-9a81-55527c2b6943/volumes"
Oct 31 20:00:04.532348 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:04.532320    4484 scope.go:122] "RemoveContainer" containerID="aee28cf83cab450dc105fb0eb001dd262db219fd7b9e6aa59e4318eddd580359"
Oct 31 20:00:04.532764 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:00:04.532594    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891280-dcpnx_odh-ai-gateway-infra(c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6"
Oct 31 20:00:10.587829 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:10.587785    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:00:16.987116 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:16.987084    4484 scope.go:122] "RemoveContainer" containerID="aee28cf83cab450dc105fb0eb001dd262db219fd7b9e6aa59e4318eddd580359"
Oct 31 20:00:17.607680 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:17.607586    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx" event={"ID":"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6","Type":"ContainerStarted","Data":"6d30d9f19831ee4381c254284070a4eb4e0b8c1cd78f61fa65f3c06a82f05c1e"}
Oct 31 20:00:17.728705 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:17.728654    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx" podStartSLOduration=17.728641256 podStartE2EDuration="17.728641256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:17.727501384 +0000 UTC m=+5963.873951238" watchObservedRunningTime="2026-10-31 20:00:17.728641256 +0000 UTC m=+5963.875091119"
Oct 31 20:00:18.615142 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:18.615109    4484 generic.go:356] "Generic (PLEG): container finished" podID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" containerID="6d30d9f19831ee4381c254284070a4eb4e0b8c1cd78f61fa65f3c06a82f05c1e" exitCode=60
Oct 31 20:00:18.615610 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:18.615180    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx" event={"ID":"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6","Type":"ContainerDied","Data":"6d30d9f19831ee4381c254284070a4eb4e0b8c1cd78f61fa65f3c06a82f05c1e"}
Oct 31 20:00:18.615610 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:18.615244    4484 scope.go:122] "RemoveContainer" containerID="aee28cf83cab450dc105fb0eb001dd262db219fd7b9e6aa59e4318eddd580359"
Oct 31 20:00:18.615705 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:18.615616    4484 scope.go:122] "RemoveContainer" containerID="6d30d9f19831ee4381c254284070a4eb4e0b8c1cd78f61fa65f3c06a82f05c1e"
Oct 31 20:00:18.615911 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:00:18.615887    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891280-dcpnx_odh-ai-gateway-infra(c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6"
Oct 31 20:00:18.739376 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:18.739344    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"]
Oct 31 20:00:19.755041 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:19.755019    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"
Oct 31 20:00:19.812799 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:19.812774    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-kube-api-access-75rk9\" (UniqueName: \"kubernetes.io/projected/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-kube-api-access-75rk9\") pod \"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6\" (UID: \"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6\") "
Oct 31 20:00:19.812954 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:19.812813    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-maas-api-service-ca\") pod \"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6\" (UID: \"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6\") "
Oct 31 20:00:19.813187 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:19.813165    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-maas-api-service-ca" pod "c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" (UID: "c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:19.814976 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:19.814956    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-kube-api-access-75rk9" pod "c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" (UID: "c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6"). InnerVolumeSpecName "kube-api-access-75rk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:19.914312 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:19.914234    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rk9\" (UniqueName: \"kubernetes.io/projected/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-kube-api-access-75rk9\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:19.914312 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:19.914263    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:20.627714 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:20.627686    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"
Oct 31 20:00:20.627714 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:20.627695    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx" event={"ID":"c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6","Type":"ContainerDied","Data":"1a93cc1f80c7844878ae9109a0d622f7eef533c848280947f4bbccde576cecd0"}
Oct 31 20:00:20.627958 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:20.627756    4484 scope.go:122] "RemoveContainer" containerID="6d30d9f19831ee4381c254284070a4eb4e0b8c1cd78f61fa65f3c06a82f05c1e"
Oct 31 20:00:20.796276 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:20.796247    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"]
Oct 31 20:00:20.834839 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:20.834808    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-dcpnx"]
Oct 31 20:00:21.993165 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:21.993132    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" path="/var/lib/kubelet/pods/c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6/volumes"
Oct 31 20:00:25.467620 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:00:25.467582    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:00:29.168384 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:29.168346    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:00:36.333235 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:36.332915    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fs5lm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:00:36.333235 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:36.332970    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm" podUID="8082f048-2aba-4b78-8f8e-dedf3015ea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:00:40.194356 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:40.194318    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:00:41.795944 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.795910    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"]
Oct 31 20:00:41.796337 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.796292    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" containerName="cleanup"
Oct 31 20:00:41.796337 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.796302    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" containerName="cleanup"
Oct 31 20:00:41.796337 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.796320    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" containerName="cleanup"
Oct 31 20:00:41.796337 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.796325    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" containerName="cleanup"
Oct 31 20:00:41.796337 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.796334    4484 cpu_manager.go:414] "containerMap: removing container" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" containerName="cleanup"
Oct 31 20:00:41.796337 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.796339    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" containerName="cleanup"
Oct 31 20:00:41.796560 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.796358    4484 cpu_manager.go:414] "containerMap: removing container" podUID="15bc65fb-d639-42b4-9a81-55527c2b6943" containerName="payload-processing"
Oct 31 20:00:41.796560 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.796363    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15bc65fb-d639-42b4-9a81-55527c2b6943" containerName="payload-processing"
Oct 31 20:00:41.796560 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.796420    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="15bc65fb-d639-42b4-9a81-55527c2b6943" containerName="payload-processing"
Oct 31 20:00:41.796560 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.796429    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" containerName="cleanup"
Oct 31 20:00:41.796560 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.796436    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" containerName="cleanup"
Oct 31 20:00:41.796560 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.796442    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d97cb1-0e6f-4149-b97c-d60d8be2e6e6" containerName="cleanup"
Oct 31 20:00:41.800825 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.800810    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:41.856713 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.856681    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"]
Oct 31 20:00:41.905854 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.905827    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:00:41.906020 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.905878    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:00:41.954860 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.954835    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzmh2\" (UniqueName: \"kubernetes.io/projected/6929e529-bf71-4487-98d3-5e484cf068b6-kube-api-access-zzmh2\") pod \"maas-api-revoke-sub-2e1f5b5f-86kbh\" (UID: \"6929e529-bf71-4487-98d3-5e484cf068b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:41.955018 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.954869    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-86kbh\" (UID: \"6929e529-bf71-4487-98d3-5e484cf068b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:41.955018 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:41.954888    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-86kbh\" (UID: \"6929e529-bf71-4487-98d3-5e484cf068b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:42.056295 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:42.056219    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzmh2\" (UniqueName: \"kubernetes.io/projected/6929e529-bf71-4487-98d3-5e484cf068b6-kube-api-access-zzmh2\") pod \"maas-api-revoke-sub-2e1f5b5f-86kbh\" (UID: \"6929e529-bf71-4487-98d3-5e484cf068b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:42.056295 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:42.056258    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-86kbh\" (UID: \"6929e529-bf71-4487-98d3-5e484cf068b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:42.056295 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:42.056280    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-86kbh\" (UID: \"6929e529-bf71-4487-98d3-5e484cf068b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:42.056892 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:42.056872    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-86kbh\" (UID: \"6929e529-bf71-4487-98d3-5e484cf068b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:42.058810 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:42.058793    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-86kbh\" (UID: \"6929e529-bf71-4487-98d3-5e484cf068b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:42.114873 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:42.114845    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzmh2\" (UniqueName: \"kubernetes.io/projected/6929e529-bf71-4487-98d3-5e484cf068b6-kube-api-access-zzmh2\") pod \"maas-api-revoke-sub-2e1f5b5f-86kbh\" (UID: \"6929e529-bf71-4487-98d3-5e484cf068b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:42.414264 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:42.414194    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:42.785607 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:42.785581    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"]
Oct 31 20:00:42.787349 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:00:42.787326    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6929e529_bf71_4487_98d3_5e484cf068b6.slice/crio-9ad1d9290c7879ce6f0d2949ec8025da1733f359ed2d5dd8d676a1efec1c2e06 WatchSource:0}: Error finding container 9ad1d9290c7879ce6f0d2949ec8025da1733f359ed2d5dd8d676a1efec1c2e06: Status 404 returned error can't find the container with id 9ad1d9290c7879ce6f0d2949ec8025da1733f359ed2d5dd8d676a1efec1c2e06
Oct 31 20:00:43.753732 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:43.753698    4484 generic.go:356] "Generic (PLEG): container finished" podID="6929e529-bf71-4487-98d3-5e484cf068b6" containerID="866cecb28958805984a6f1c921422d5a0a3b1265f76f407f5c48f7ffdb544f01" exitCode=0
Oct 31 20:00:43.754111 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:43.753740    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh" event={"ID":"6929e529-bf71-4487-98d3-5e484cf068b6","Type":"ContainerDied","Data":"866cecb28958805984a6f1c921422d5a0a3b1265f76f407f5c48f7ffdb544f01"}
Oct 31 20:00:43.754111 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:43.753786    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh" event={"ID":"6929e529-bf71-4487-98d3-5e484cf068b6","Type":"ContainerStarted","Data":"9ad1d9290c7879ce6f0d2949ec8025da1733f359ed2d5dd8d676a1efec1c2e06"}
Oct 31 20:00:44.893094 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:44.893069    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:44.986364 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:44.986335    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-xks-ca\") pod \"6929e529-bf71-4487-98d3-5e484cf068b6\" (UID: \"6929e529-bf71-4487-98d3-5e484cf068b6\") "
Oct 31 20:00:44.986545 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:44.986451    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6929e529-bf71-4487-98d3-5e484cf068b6-kube-api-access-zzmh2\" (UniqueName: \"kubernetes.io/projected/6929e529-bf71-4487-98d3-5e484cf068b6-kube-api-access-zzmh2\") pod \"6929e529-bf71-4487-98d3-5e484cf068b6\" (UID: \"6929e529-bf71-4487-98d3-5e484cf068b6\") "
Oct 31 20:00:44.986656 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:44.986634    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-openshift-ca\") pod \"6929e529-bf71-4487-98d3-5e484cf068b6\" (UID: \"6929e529-bf71-4487-98d3-5e484cf068b6\") "
Oct 31 20:00:44.986996 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:44.986971    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-openshift-ca" pod "6929e529-bf71-4487-98d3-5e484cf068b6" (UID: "6929e529-bf71-4487-98d3-5e484cf068b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:44.988617 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:44.988593    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6929e529-bf71-4487-98d3-5e484cf068b6-kube-api-access-zzmh2" pod "6929e529-bf71-4487-98d3-5e484cf068b6" (UID: "6929e529-bf71-4487-98d3-5e484cf068b6"). InnerVolumeSpecName "kube-api-access-zzmh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:44.988617 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:44.988602    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-xks-ca" pod "6929e529-bf71-4487-98d3-5e484cf068b6" (UID: "6929e529-bf71-4487-98d3-5e484cf068b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:45.088099 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:45.088009    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:45.088099 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:45.088046    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzmh2\" (UniqueName: \"kubernetes.io/projected/6929e529-bf71-4487-98d3-5e484cf068b6-kube-api-access-zzmh2\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:45.088099 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:45.088061    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6929e529-bf71-4487-98d3-5e484cf068b6-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:45.765582 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:45.765548    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh" event={"ID":"6929e529-bf71-4487-98d3-5e484cf068b6","Type":"ContainerDied","Data":"9ad1d9290c7879ce6f0d2949ec8025da1733f359ed2d5dd8d676a1efec1c2e06"}
Oct 31 20:00:45.765582 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:45.765567    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"
Oct 31 20:00:45.765795 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:45.765597    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ad1d9290c7879ce6f0d2949ec8025da1733f359ed2d5dd8d676a1efec1c2e06"
Oct 31 20:00:54.266142 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.266107    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"]
Oct 31 20:00:54.266591 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.266577    4484 cpu_manager.go:414] "containerMap: removing container" podUID="6929e529-bf71-4487-98d3-5e484cf068b6" containerName="revoke-keys"
Oct 31 20:00:54.266640 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.266595    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6929e529-bf71-4487-98d3-5e484cf068b6" containerName="revoke-keys"
Oct 31 20:00:54.266679 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.266657    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="6929e529-bf71-4487-98d3-5e484cf068b6" containerName="revoke-keys"
Oct 31 20:00:54.269854 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.269836    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:54.327794 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.327763    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"]
Oct 31 20:00:54.369254 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.369186    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dntfj\" (UID: \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:54.369424 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.369370    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmnqp\" (UniqueName: \"kubernetes.io/projected/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-kube-api-access-xmnqp\") pod \"maas-api-revoke-sub-ec843018-dntfj\" (UID: \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:54.369424 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.369408    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dntfj\" (UID: \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:54.375647 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.375630    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:00:54.375647 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.375633    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:00:54.469949 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.469918    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dntfj\" (UID: \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:54.470115 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.469985    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmnqp\" (UniqueName: \"kubernetes.io/projected/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-kube-api-access-xmnqp\") pod \"maas-api-revoke-sub-ec843018-dntfj\" (UID: \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:54.470115 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.470022    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dntfj\" (UID: \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:54.470726 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.470697    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dntfj\" (UID: \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:54.472346 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.472317    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dntfj\" (UID: \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:54.528015 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.527948    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmnqp\" (UniqueName: \"kubernetes.io/projected/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-kube-api-access-xmnqp\") pod \"maas-api-revoke-sub-ec843018-dntfj\" (UID: \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:54.582612 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.582584    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:54.695932 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.695902    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:00:54.953637 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:54.953613    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"]
Oct 31 20:00:54.955600 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:00:54.955572    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc510fa5_8ea4_45d2_ad15_a8f78f8a3884.slice/crio-a70e447704a4a63556b2213d552f8b04730c86a8a85fb5494c9f5038c72070e8 WatchSource:0}: Error finding container a70e447704a4a63556b2213d552f8b04730c86a8a85fb5494c9f5038c72070e8: Status 404 returned error can't find the container with id a70e447704a4a63556b2213d552f8b04730c86a8a85fb5494c9f5038c72070e8
Oct 31 20:00:55.053236 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:55.053205    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 20:00:55.060432 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:55.060410    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 20:00:55.467596 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:00:55.467561    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:00:55.812628 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:55.812597    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj" event={"ID":"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884","Type":"ContainerStarted","Data":"e4bbf35b6ece31f239f2f3ee2f88ec63ae834147a7cb861910e9cbf8420f1b5f"}
Oct 31 20:00:55.812820 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:55.812649    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj" event={"ID":"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884","Type":"ContainerStarted","Data":"a70e447704a4a63556b2213d552f8b04730c86a8a85fb5494c9f5038c72070e8"}
Oct 31 20:00:55.930018 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:55.929965    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj" podStartSLOduration=1.92995032 podStartE2EDuration="1.92995032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:55.9282306 +0000 UTC m=+6002.074680465" watchObservedRunningTime="2026-10-31 20:00:55.92995032 +0000 UTC m=+6002.076400183"
Oct 31 20:00:56.818743 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:56.818708    4484 generic.go:356] "Generic (PLEG): container finished" podID="cc510fa5-8ea4-45d2-ad15-a8f78f8a3884" containerID="e4bbf35b6ece31f239f2f3ee2f88ec63ae834147a7cb861910e9cbf8420f1b5f" exitCode=0
Oct 31 20:00:56.819140 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:56.818759    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj" event={"ID":"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884","Type":"ContainerDied","Data":"e4bbf35b6ece31f239f2f3ee2f88ec63ae834147a7cb861910e9cbf8420f1b5f"}
Oct 31 20:00:57.956851 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:57.956828    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:58.005248 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:58.005224    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-kube-api-access-xmnqp\" (UniqueName: \"kubernetes.io/projected/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-kube-api-access-xmnqp\") pod \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\" (UID: \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\") "
Oct 31 20:00:58.005464 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:58.005265    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-openshift-ca\") pod \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\" (UID: \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\") "
Oct 31 20:00:58.005464 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:58.005390    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-xks-ca\") pod \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\" (UID: \"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884\") "
Oct 31 20:00:58.005700 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:58.005673    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-openshift-ca" pod "cc510fa5-8ea4-45d2-ad15-a8f78f8a3884" (UID: "cc510fa5-8ea4-45d2-ad15-a8f78f8a3884"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:58.007412 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:58.007386    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-xks-ca" pod "cc510fa5-8ea4-45d2-ad15-a8f78f8a3884" (UID: "cc510fa5-8ea4-45d2-ad15-a8f78f8a3884"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:58.007552 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:58.007389    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-kube-api-access-xmnqp" pod "cc510fa5-8ea4-45d2-ad15-a8f78f8a3884" (UID: "cc510fa5-8ea4-45d2-ad15-a8f78f8a3884"). InnerVolumeSpecName "kube-api-access-xmnqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:58.106229 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:58.106160    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:58.106229 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:58.106186    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmnqp\" (UniqueName: \"kubernetes.io/projected/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-kube-api-access-xmnqp\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:58.106229 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:58.106196    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:00:58.830490 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:58.830449    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"
Oct 31 20:00:58.830669 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:58.830444    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj" event={"ID":"cc510fa5-8ea4-45d2-ad15-a8f78f8a3884","Type":"ContainerDied","Data":"a70e447704a4a63556b2213d552f8b04730c86a8a85fb5494c9f5038c72070e8"}
Oct 31 20:00:58.830669 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:00:58.830580    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a70e447704a4a63556b2213d552f8b04730c86a8a85fb5494c9f5038c72070e8"
Oct 31 20:01:11.480433 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:11.480355    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:01:12.048398 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:12.048355    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"]
Oct 31 20:01:12.097962 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:12.097928    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wfn7l"]
Oct 31 20:01:14.025500 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:14.019416    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d92140-c003-46e2-9717-9ddc11015581" path="/var/lib/kubelet/pods/48d92140-c003-46e2-9717-9ddc11015581/volumes"
Oct 31 20:01:25.466876 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:01:25.466835    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:01:29.593914 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.593876    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"]
Oct 31 20:01:29.594304 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.594283    4484 cpu_manager.go:414] "containerMap: removing container" podUID="cc510fa5-8ea4-45d2-ad15-a8f78f8a3884" containerName="revoke-keys"
Oct 31 20:01:29.594304 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.594298    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc510fa5-8ea4-45d2-ad15-a8f78f8a3884" containerName="revoke-keys"
Oct 31 20:01:29.594422 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.594411    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc510fa5-8ea4-45d2-ad15-a8f78f8a3884" containerName="revoke-keys"
Oct 31 20:01:29.597446 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.597429    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:29.702900 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.702871    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:01:29.702900 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.702880    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:01:29.705061 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.705038    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"]
Oct 31 20:01:29.729104 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.729076    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-tztzw\" (UID: \"d08ea57a-6133-4de8-9827-53adfadac0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:29.729245 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.729152    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr7zb\" (UniqueName: \"kubernetes.io/projected/d08ea57a-6133-4de8-9827-53adfadac0c1-kube-api-access-qr7zb\") pod \"maas-api-revoke-sub-046e1343-tztzw\" (UID: \"d08ea57a-6133-4de8-9827-53adfadac0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:29.729245 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.729231    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-tztzw\" (UID: \"d08ea57a-6133-4de8-9827-53adfadac0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:29.830092 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.830060    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-tztzw\" (UID: \"d08ea57a-6133-4de8-9827-53adfadac0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:29.830249 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.830118    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7zb\" (UniqueName: \"kubernetes.io/projected/d08ea57a-6133-4de8-9827-53adfadac0c1-kube-api-access-qr7zb\") pod \"maas-api-revoke-sub-046e1343-tztzw\" (UID: \"d08ea57a-6133-4de8-9827-53adfadac0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:29.830249 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.830149    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-tztzw\" (UID: \"d08ea57a-6133-4de8-9827-53adfadac0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:29.830832 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.830809    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-tztzw\" (UID: \"d08ea57a-6133-4de8-9827-53adfadac0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:29.832544 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.832518    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-tztzw\" (UID: \"d08ea57a-6133-4de8-9827-53adfadac0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:29.887859 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.887803    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7zb\" (UniqueName: \"kubernetes.io/projected/d08ea57a-6133-4de8-9827-53adfadac0c1-kube-api-access-qr7zb\") pod \"maas-api-revoke-sub-046e1343-tztzw\" (UID: \"d08ea57a-6133-4de8-9827-53adfadac0c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:29.910679 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:29.910654    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:30.279717 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:30.279677    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"]
Oct 31 20:01:30.280969 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:01:30.280947    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08ea57a_6133_4de8_9827_53adfadac0c1.slice/crio-4783ceae3aa78a68f34517003d73a862121c3931698de0a0755cc845b97861a5 WatchSource:0}: Error finding container 4783ceae3aa78a68f34517003d73a862121c3931698de0a0755cc845b97861a5: Status 404 returned error can't find the container with id 4783ceae3aa78a68f34517003d73a862121c3931698de0a0755cc845b97861a5
Oct 31 20:01:31.003187 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:31.003148    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw" event={"ID":"d08ea57a-6133-4de8-9827-53adfadac0c1","Type":"ContainerStarted","Data":"323df548e31866dd076ddeb260c1c2ee35be0963862423c897b2d49abc351e06"}
Oct 31 20:01:31.003704 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:31.003229    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw" event={"ID":"d08ea57a-6133-4de8-9827-53adfadac0c1","Type":"ContainerStarted","Data":"4783ceae3aa78a68f34517003d73a862121c3931698de0a0755cc845b97861a5"}
Oct 31 20:01:31.118993 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:31.118946    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw" podStartSLOduration=2.118934166 podStartE2EDuration="2.118934166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:31.118227035 +0000 UTC m=+6037.264676901" watchObservedRunningTime="2026-10-31 20:01:31.118934166 +0000 UTC m=+6037.265384029"
Oct 31 20:01:32.009608 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:32.009571    4484 generic.go:356] "Generic (PLEG): container finished" podID="d08ea57a-6133-4de8-9827-53adfadac0c1" containerID="323df548e31866dd076ddeb260c1c2ee35be0963862423c897b2d49abc351e06" exitCode=0
Oct 31 20:01:32.010015 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:32.009625    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw" event={"ID":"d08ea57a-6133-4de8-9827-53adfadac0c1","Type":"ContainerDied","Data":"323df548e31866dd076ddeb260c1c2ee35be0963862423c897b2d49abc351e06"}
Oct 31 20:01:33.146673 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:33.146649    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:33.261377 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:33.261345    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-xks-ca\") pod \"d08ea57a-6133-4de8-9827-53adfadac0c1\" (UID: \"d08ea57a-6133-4de8-9827-53adfadac0c1\") "
Oct 31 20:01:33.261553 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:33.261412    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d08ea57a-6133-4de8-9827-53adfadac0c1-kube-api-access-qr7zb\" (UniqueName: \"kubernetes.io/projected/d08ea57a-6133-4de8-9827-53adfadac0c1-kube-api-access-qr7zb\") pod \"d08ea57a-6133-4de8-9827-53adfadac0c1\" (UID: \"d08ea57a-6133-4de8-9827-53adfadac0c1\") "
Oct 31 20:01:33.261553 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:33.261511    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-openshift-ca\") pod \"d08ea57a-6133-4de8-9827-53adfadac0c1\" (UID: \"d08ea57a-6133-4de8-9827-53adfadac0c1\") "
Oct 31 20:01:33.261928 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:33.261858    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-openshift-ca" pod "d08ea57a-6133-4de8-9827-53adfadac0c1" (UID: "d08ea57a-6133-4de8-9827-53adfadac0c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:33.264035 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:33.263979    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08ea57a-6133-4de8-9827-53adfadac0c1-kube-api-access-qr7zb" pod "d08ea57a-6133-4de8-9827-53adfadac0c1" (UID: "d08ea57a-6133-4de8-9827-53adfadac0c1"). InnerVolumeSpecName "kube-api-access-qr7zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:33.264184 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:33.264128    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-xks-ca" pod "d08ea57a-6133-4de8-9827-53adfadac0c1" (UID: "d08ea57a-6133-4de8-9827-53adfadac0c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:33.363021 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:33.362987    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:01:33.363021 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:33.363019    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08ea57a-6133-4de8-9827-53adfadac0c1-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:01:33.363021 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:33.363030    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr7zb\" (UniqueName: \"kubernetes.io/projected/d08ea57a-6133-4de8-9827-53adfadac0c1-kube-api-access-qr7zb\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:01:34.021712 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:34.021682    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"
Oct 31 20:01:34.021862 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:34.021710    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw" event={"ID":"d08ea57a-6133-4de8-9827-53adfadac0c1","Type":"ContainerDied","Data":"4783ceae3aa78a68f34517003d73a862121c3931698de0a0755cc845b97861a5"}
Oct 31 20:01:34.021862 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:34.021754    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4783ceae3aa78a68f34517003d73a862121c3931698de0a0755cc845b97861a5"
Oct 31 20:01:40.345358 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.345326    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"]
Oct 31 20:01:40.345787 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.345773    4484 cpu_manager.go:414] "containerMap: removing container" podUID="d08ea57a-6133-4de8-9827-53adfadac0c1" containerName="revoke-keys"
Oct 31 20:01:40.345836 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.345789    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d08ea57a-6133-4de8-9827-53adfadac0c1" containerName="revoke-keys"
Oct 31 20:01:40.345876 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.345858    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="d08ea57a-6133-4de8-9827-53adfadac0c1" containerName="revoke-keys"
Oct 31 20:01:40.349105 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.349070    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:40.405119 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.405083    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"]
Oct 31 20:01:40.429972 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.429940    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bl9t6\" (UID: \"9bf2bc07-7df8-4120-a99f-6639134a5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:40.430133 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.429989    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bl9t6\" (UID: \"9bf2bc07-7df8-4120-a99f-6639134a5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:40.430179 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.430142    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jzl6\" (UniqueName: \"kubernetes.io/projected/9bf2bc07-7df8-4120-a99f-6639134a5279-kube-api-access-8jzl6\") pod \"maas-api-revoke-sub-d3fba056-bl9t6\" (UID: \"9bf2bc07-7df8-4120-a99f-6639134a5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:40.453574 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.453544    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:01:40.453746 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.453591    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:01:40.531672 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.531640    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bl9t6\" (UID: \"9bf2bc07-7df8-4120-a99f-6639134a5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:40.531841 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.531684    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bl9t6\" (UID: \"9bf2bc07-7df8-4120-a99f-6639134a5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:40.531841 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.531732    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jzl6\" (UniqueName: \"kubernetes.io/projected/9bf2bc07-7df8-4120-a99f-6639134a5279-kube-api-access-8jzl6\") pod \"maas-api-revoke-sub-d3fba056-bl9t6\" (UID: \"9bf2bc07-7df8-4120-a99f-6639134a5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:40.532280 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.532259    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-bl9t6\" (UID: \"9bf2bc07-7df8-4120-a99f-6639134a5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:40.534201 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.534177    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-bl9t6\" (UID: \"9bf2bc07-7df8-4120-a99f-6639134a5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:40.589943 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.589916    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jzl6\" (UniqueName: \"kubernetes.io/projected/9bf2bc07-7df8-4120-a99f-6639134a5279-kube-api-access-8jzl6\") pod \"maas-api-revoke-sub-d3fba056-bl9t6\" (UID: \"9bf2bc07-7df8-4120-a99f-6639134a5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:40.662172 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:40.662108    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:41.034012 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:41.033982    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"]
Oct 31 20:01:41.035633 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:01:41.035609    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf2bc07_7df8_4120_a99f_6639134a5279.slice/crio-3d46fa777f3f8ef5df43948b6288d7758b8be951cac4e601cf70d8b5df8b5522 WatchSource:0}: Error finding container 3d46fa777f3f8ef5df43948b6288d7758b8be951cac4e601cf70d8b5df8b5522: Status 404 returned error can't find the container with id 3d46fa777f3f8ef5df43948b6288d7758b8be951cac4e601cf70d8b5df8b5522
Oct 31 20:01:41.055868 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:41.055840    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6" event={"ID":"9bf2bc07-7df8-4120-a99f-6639134a5279","Type":"ContainerStarted","Data":"3d46fa777f3f8ef5df43948b6288d7758b8be951cac4e601cf70d8b5df8b5522"}
Oct 31 20:01:42.062305 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:42.062267    4484 generic.go:356] "Generic (PLEG): container finished" podID="9bf2bc07-7df8-4120-a99f-6639134a5279" containerID="c9349c428f397726233aaa5684afa105f0d7a2a254678fe10efa522684141e4d" exitCode=0
Oct 31 20:01:42.062708 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:42.062349    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6" event={"ID":"9bf2bc07-7df8-4120-a99f-6639134a5279","Type":"ContainerDied","Data":"c9349c428f397726233aaa5684afa105f0d7a2a254678fe10efa522684141e4d"}
Oct 31 20:01:42.496337 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:42.496293    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:01:43.222118 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:43.222093    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:43.258644 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:43.258610    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf2bc07-7df8-4120-a99f-6639134a5279-kube-api-access-8jzl6\" (UniqueName: \"kubernetes.io/projected/9bf2bc07-7df8-4120-a99f-6639134a5279-kube-api-access-8jzl6\") pod \"9bf2bc07-7df8-4120-a99f-6639134a5279\" (UID: \"9bf2bc07-7df8-4120-a99f-6639134a5279\") "
Oct 31 20:01:43.258821 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:43.258695    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-xks-ca\") pod \"9bf2bc07-7df8-4120-a99f-6639134a5279\" (UID: \"9bf2bc07-7df8-4120-a99f-6639134a5279\") "
Oct 31 20:01:43.258821 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:43.258761    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-openshift-ca\") pod \"9bf2bc07-7df8-4120-a99f-6639134a5279\" (UID: \"9bf2bc07-7df8-4120-a99f-6639134a5279\") "
Oct 31 20:01:43.259195 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:43.259160    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-openshift-ca" pod "9bf2bc07-7df8-4120-a99f-6639134a5279" (UID: "9bf2bc07-7df8-4120-a99f-6639134a5279"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:43.260913 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:43.260885    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-xks-ca" pod "9bf2bc07-7df8-4120-a99f-6639134a5279" (UID: "9bf2bc07-7df8-4120-a99f-6639134a5279"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:43.261202 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:43.261180    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf2bc07-7df8-4120-a99f-6639134a5279-kube-api-access-8jzl6" pod "9bf2bc07-7df8-4120-a99f-6639134a5279" (UID: "9bf2bc07-7df8-4120-a99f-6639134a5279"). InnerVolumeSpecName "kube-api-access-8jzl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:43.360152 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:43.360072    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jzl6\" (UniqueName: \"kubernetes.io/projected/9bf2bc07-7df8-4120-a99f-6639134a5279-kube-api-access-8jzl6\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:01:43.360152 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:43.360116    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:01:43.360152 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:43.360133    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf2bc07-7df8-4120-a99f-6639134a5279-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:01:44.099498 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:44.098995    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6" event={"ID":"9bf2bc07-7df8-4120-a99f-6639134a5279","Type":"ContainerDied","Data":"3d46fa777f3f8ef5df43948b6288d7758b8be951cac4e601cf70d8b5df8b5522"}
Oct 31 20:01:44.099498 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:44.099060    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d46fa777f3f8ef5df43948b6288d7758b8be951cac4e601cf70d8b5df8b5522"
Oct 31 20:01:44.099498 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:44.099130    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"
Oct 31 20:01:53.087162 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:01:53.087123    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:01:55.467208 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:01:55.467172    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:02:05.046057 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:05.046016    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"]
Oct 31 20:02:05.096992 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:05.096957    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f166acd-wwxv4"]
Oct 31 20:02:05.862352 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:05.862267    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:02:05.992293 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:05.992254    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29d5139-b34b-44c5-803d-49ad53f1609d" path="/var/lib/kubelet/pods/a29d5139-b34b-44c5-803d-49ad53f1609d/volumes"
Oct 31 20:02:15.222358 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.222316    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"]
Oct 31 20:02:15.222818 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.222802    4484 cpu_manager.go:414] "containerMap: removing container" podUID="9bf2bc07-7df8-4120-a99f-6639134a5279" containerName="revoke-keys"
Oct 31 20:02:15.222862 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.222818    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf2bc07-7df8-4120-a99f-6639134a5279" containerName="revoke-keys"
Oct 31 20:02:15.222911 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.222898    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf2bc07-7df8-4120-a99f-6639134a5279" containerName="revoke-keys"
Oct 31 20:02:15.227877 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.227858    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:15.282322 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.282286    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"]
Oct 31 20:02:15.330873 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.330842    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:02:15.331137 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.331118    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:02:15.361436 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.361399    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-hmc7x\" (UID: \"0d883f9f-105c-4e12-a86c-1e31216c932b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:15.361661 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.361444    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-hmc7x\" (UID: \"0d883f9f-105c-4e12-a86c-1e31216c932b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:15.361661 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.361588    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrr2\" (UniqueName: \"kubernetes.io/projected/0d883f9f-105c-4e12-a86c-1e31216c932b-kube-api-access-rqrr2\") pod \"maas-api-revoke-sub-ae594979-hmc7x\" (UID: \"0d883f9f-105c-4e12-a86c-1e31216c932b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:15.462217 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.462174    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrr2\" (UniqueName: \"kubernetes.io/projected/0d883f9f-105c-4e12-a86c-1e31216c932b-kube-api-access-rqrr2\") pod \"maas-api-revoke-sub-ae594979-hmc7x\" (UID: \"0d883f9f-105c-4e12-a86c-1e31216c932b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:15.462440 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.462285    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-hmc7x\" (UID: \"0d883f9f-105c-4e12-a86c-1e31216c932b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:15.462440 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.462316    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-hmc7x\" (UID: \"0d883f9f-105c-4e12-a86c-1e31216c932b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:15.462945 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.462924    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-hmc7x\" (UID: \"0d883f9f-105c-4e12-a86c-1e31216c932b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:15.464978 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.464947    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-hmc7x\" (UID: \"0d883f9f-105c-4e12-a86c-1e31216c932b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:15.521950 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.521911    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrr2\" (UniqueName: \"kubernetes.io/projected/0d883f9f-105c-4e12-a86c-1e31216c932b-kube-api-access-rqrr2\") pod \"maas-api-revoke-sub-ae594979-hmc7x\" (UID: \"0d883f9f-105c-4e12-a86c-1e31216c932b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:15.541606 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.541573    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:15.916711 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:15.916686    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"]
Oct 31 20:02:15.918699 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:02:15.918672    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d883f9f_105c_4e12_a86c_1e31216c932b.slice/crio-7dd93f5c3479d63d84c05054e06bfa2b169a9f43d20c7565cd8c373e9c7b7f2d WatchSource:0}: Error finding container 7dd93f5c3479d63d84c05054e06bfa2b169a9f43d20c7565cd8c373e9c7b7f2d: Status 404 returned error can't find the container with id 7dd93f5c3479d63d84c05054e06bfa2b169a9f43d20c7565cd8c373e9c7b7f2d
Oct 31 20:02:16.294201 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:16.294162    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x" event={"ID":"0d883f9f-105c-4e12-a86c-1e31216c932b","Type":"ContainerStarted","Data":"52b769812de45a127841d4ba8899da1690667fba9cb1478bffc5ccece70f0851"}
Oct 31 20:02:16.294713 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:16.294232    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x" event={"ID":"0d883f9f-105c-4e12-a86c-1e31216c932b","Type":"ContainerStarted","Data":"7dd93f5c3479d63d84c05054e06bfa2b169a9f43d20c7565cd8c373e9c7b7f2d"}
Oct 31 20:02:16.413012 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:16.412950    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x" podStartSLOduration=1.412935443 podStartE2EDuration="1.412935443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:16.41072881 +0000 UTC m=+6082.557178677" watchObservedRunningTime="2026-10-31 20:02:16.412935443 +0000 UTC m=+6082.559385326"
Oct 31 20:02:17.301218 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:17.301181    4484 generic.go:356] "Generic (PLEG): container finished" podID="0d883f9f-105c-4e12-a86c-1e31216c932b" containerID="52b769812de45a127841d4ba8899da1690667fba9cb1478bffc5ccece70f0851" exitCode=0
Oct 31 20:02:17.301696 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:17.301279    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x" event={"ID":"0d883f9f-105c-4e12-a86c-1e31216c932b","Type":"ContainerDied","Data":"52b769812de45a127841d4ba8899da1690667fba9cb1478bffc5ccece70f0851"}
Oct 31 20:02:18.448901 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:18.448873    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:18.593648 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:18.593565    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-openshift-ca\") pod \"0d883f9f-105c-4e12-a86c-1e31216c932b\" (UID: \"0d883f9f-105c-4e12-a86c-1e31216c932b\") "
Oct 31 20:02:18.593826 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:18.593671    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-xks-ca\") pod \"0d883f9f-105c-4e12-a86c-1e31216c932b\" (UID: \"0d883f9f-105c-4e12-a86c-1e31216c932b\") "
Oct 31 20:02:18.593826 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:18.593812    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d883f9f-105c-4e12-a86c-1e31216c932b-kube-api-access-rqrr2\" (UniqueName: \"kubernetes.io/projected/0d883f9f-105c-4e12-a86c-1e31216c932b-kube-api-access-rqrr2\") pod \"0d883f9f-105c-4e12-a86c-1e31216c932b\" (UID: \"0d883f9f-105c-4e12-a86c-1e31216c932b\") "
Oct 31 20:02:18.594028 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:18.594003    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-openshift-ca" pod "0d883f9f-105c-4e12-a86c-1e31216c932b" (UID: "0d883f9f-105c-4e12-a86c-1e31216c932b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:18.594213 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:18.594185    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:02:18.596024 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:18.595997    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-xks-ca" pod "0d883f9f-105c-4e12-a86c-1e31216c932b" (UID: "0d883f9f-105c-4e12-a86c-1e31216c932b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:18.596172 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:18.596124    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d883f9f-105c-4e12-a86c-1e31216c932b-kube-api-access-rqrr2" pod "0d883f9f-105c-4e12-a86c-1e31216c932b" (UID: "0d883f9f-105c-4e12-a86c-1e31216c932b"). InnerVolumeSpecName "kube-api-access-rqrr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:18.695547 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:18.695505    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d883f9f-105c-4e12-a86c-1e31216c932b-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:02:18.695547 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:18.695541    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqrr2\" (UniqueName: \"kubernetes.io/projected/0d883f9f-105c-4e12-a86c-1e31216c932b-kube-api-access-rqrr2\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:02:19.313271 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:19.313244    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"
Oct 31 20:02:19.313468 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:19.313286    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x" event={"ID":"0d883f9f-105c-4e12-a86c-1e31216c932b","Type":"ContainerDied","Data":"7dd93f5c3479d63d84c05054e06bfa2b169a9f43d20c7565cd8c373e9c7b7f2d"}
Oct 31 20:02:19.313468 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:19.313341    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dd93f5c3479d63d84c05054e06bfa2b169a9f43d20c7565cd8c373e9c7b7f2d"
Oct 31 20:02:25.470289 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:02:25.470248    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:02:28.062966 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:28.062928    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:02:33.918578 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:33.903463    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"]
Oct 31 20:02:33.919927 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:33.919895    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0d883f9f-105c-4e12-a86c-1e31216c932b" containerName="revoke-keys"
Oct 31 20:02:33.920122 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:33.920105    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d883f9f-105c-4e12-a86c-1e31216c932b" containerName="revoke-keys"
Oct 31 20:02:33.920388 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:33.920370    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d883f9f-105c-4e12-a86c-1e31216c932b" containerName="revoke-keys"
Oct 31 20:02:33.924677 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:33.924645    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:34.043234 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.043193    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:02:34.043575 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.043553    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:02:34.056316 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.056181    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2z4vd\" (UID: \"3bc76d40-1a8f-4314-a90e-953c5ad56414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:34.056536 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.056364    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4th\" (UniqueName: \"kubernetes.io/projected/3bc76d40-1a8f-4314-a90e-953c5ad56414-kube-api-access-5b4th\") pod \"maas-api-revoke-sub-a4aef7d7-2z4vd\" (UID: \"3bc76d40-1a8f-4314-a90e-953c5ad56414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:34.056536 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.056496    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2z4vd\" (UID: \"3bc76d40-1a8f-4314-a90e-953c5ad56414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:34.059869 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.059835    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"]
Oct 31 20:02:34.157313 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.157264    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2z4vd\" (UID: \"3bc76d40-1a8f-4314-a90e-953c5ad56414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:34.157732 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.157690    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2z4vd\" (UID: \"3bc76d40-1a8f-4314-a90e-953c5ad56414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:34.157872 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.157752    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4th\" (UniqueName: \"kubernetes.io/projected/3bc76d40-1a8f-4314-a90e-953c5ad56414-kube-api-access-5b4th\") pod \"maas-api-revoke-sub-a4aef7d7-2z4vd\" (UID: \"3bc76d40-1a8f-4314-a90e-953c5ad56414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:34.158280 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.158256    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2z4vd\" (UID: \"3bc76d40-1a8f-4314-a90e-953c5ad56414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:34.160347 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.160322    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2z4vd\" (UID: \"3bc76d40-1a8f-4314-a90e-953c5ad56414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:34.240278 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.240240    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4th\" (UniqueName: \"kubernetes.io/projected/3bc76d40-1a8f-4314-a90e-953c5ad56414-kube-api-access-5b4th\") pod \"maas-api-revoke-sub-a4aef7d7-2z4vd\" (UID: \"3bc76d40-1a8f-4314-a90e-953c5ad56414\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:34.348867 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.348825    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:34.867717 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:34.867554    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"]
Oct 31 20:02:35.435510 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:35.433307    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd" event={"ID":"3bc76d40-1a8f-4314-a90e-953c5ad56414","Type":"ContainerStarted","Data":"1d2d10bd59a4ec6f1e77ece32810c0f9e1f1f51098f94d68ca0f593fdc69119f"}
Oct 31 20:02:35.435510 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:35.433362    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd" event={"ID":"3bc76d40-1a8f-4314-a90e-953c5ad56414","Type":"ContainerStarted","Data":"5b2c932d435e0fb86daab74826ae587123daf26adedbdfcd142866fb48a3bed7"}
Oct 31 20:02:35.552844 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:35.552729    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd" podStartSLOduration=2.552710715 podStartE2EDuration="2.552710715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:35.549047524 +0000 UTC m=+6101.695497389" watchObservedRunningTime="2026-10-31 20:02:35.552710715 +0000 UTC m=+6101.699160581"
Oct 31 20:02:36.439457 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:36.439419    4484 generic.go:356] "Generic (PLEG): container finished" podID="3bc76d40-1a8f-4314-a90e-953c5ad56414" containerID="1d2d10bd59a4ec6f1e77ece32810c0f9e1f1f51098f94d68ca0f593fdc69119f" exitCode=0
Oct 31 20:02:36.439943 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:36.439509    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd" event={"ID":"3bc76d40-1a8f-4314-a90e-953c5ad56414","Type":"ContainerDied","Data":"1d2d10bd59a4ec6f1e77ece32810c0f9e1f1f51098f94d68ca0f593fdc69119f"}
Oct 31 20:02:37.590626 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:37.590596    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:37.621973 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:37.621937    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc76d40-1a8f-4314-a90e-953c5ad56414-kube-api-access-5b4th\" (UniqueName: \"kubernetes.io/projected/3bc76d40-1a8f-4314-a90e-953c5ad56414-kube-api-access-5b4th\") pod \"3bc76d40-1a8f-4314-a90e-953c5ad56414\" (UID: \"3bc76d40-1a8f-4314-a90e-953c5ad56414\") "
Oct 31 20:02:37.622187 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:37.621986    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-xks-ca\") pod \"3bc76d40-1a8f-4314-a90e-953c5ad56414\" (UID: \"3bc76d40-1a8f-4314-a90e-953c5ad56414\") "
Oct 31 20:02:37.622187 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:37.622013    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-openshift-ca\") pod \"3bc76d40-1a8f-4314-a90e-953c5ad56414\" (UID: \"3bc76d40-1a8f-4314-a90e-953c5ad56414\") "
Oct 31 20:02:37.622525 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:37.622463    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-openshift-ca" pod "3bc76d40-1a8f-4314-a90e-953c5ad56414" (UID: "3bc76d40-1a8f-4314-a90e-953c5ad56414"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:37.624358 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:37.624338    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-xks-ca" pod "3bc76d40-1a8f-4314-a90e-953c5ad56414" (UID: "3bc76d40-1a8f-4314-a90e-953c5ad56414"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:37.624456 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:37.624437    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc76d40-1a8f-4314-a90e-953c5ad56414-kube-api-access-5b4th" pod "3bc76d40-1a8f-4314-a90e-953c5ad56414" (UID: "3bc76d40-1a8f-4314-a90e-953c5ad56414"). InnerVolumeSpecName "kube-api-access-5b4th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:37.697310 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:37.697212    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:02:37.722980 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:37.722946    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b4th\" (UniqueName: \"kubernetes.io/projected/3bc76d40-1a8f-4314-a90e-953c5ad56414-kube-api-access-5b4th\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.722980 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:37.722978    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:02:37.723188 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:37.722989    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc76d40-1a8f-4314-a90e-953c5ad56414-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:02:38.464944 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:38.464851    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd" event={"ID":"3bc76d40-1a8f-4314-a90e-953c5ad56414","Type":"ContainerDied","Data":"5b2c932d435e0fb86daab74826ae587123daf26adedbdfcd142866fb48a3bed7"}
Oct 31 20:02:38.464944 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:38.464905    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b2c932d435e0fb86daab74826ae587123daf26adedbdfcd142866fb48a3bed7"
Oct 31 20:02:38.464944 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:38.464865    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"
Oct 31 20:02:46.059458 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:46.059422    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:02:55.070508 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.068583    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"]
Oct 31 20:02:55.070508 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.069277    4484 cpu_manager.go:414] "containerMap: removing container" podUID="3bc76d40-1a8f-4314-a90e-953c5ad56414" containerName="revoke-keys"
Oct 31 20:02:55.070508 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.069307    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc76d40-1a8f-4314-a90e-953c5ad56414" containerName="revoke-keys"
Oct 31 20:02:55.070508 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.069437    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc76d40-1a8f-4314-a90e-953c5ad56414" containerName="revoke-keys"
Oct 31 20:02:55.102504 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.093794    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:55.146507 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.146028    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"]
Oct 31 20:02:55.184508 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.182140    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:02:55.184508 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.182329    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:02:55.251502 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.248161    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7srhw\" (UniqueName: \"kubernetes.io/projected/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-kube-api-access-7srhw\") pod \"maas-api-revoke-sub-79805ca9-kgv8g\" (UID: \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:55.251502 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.248220    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-kgv8g\" (UID: \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:55.251502 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.248312    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-kgv8g\" (UID: \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:55.349659 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.349573    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7srhw\" (UniqueName: \"kubernetes.io/projected/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-kube-api-access-7srhw\") pod \"maas-api-revoke-sub-79805ca9-kgv8g\" (UID: \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:55.349659 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.349613    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-kgv8g\" (UID: \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:55.349855 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.349690    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-kgv8g\" (UID: \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:55.350611 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.350548    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-kgv8g\" (UID: \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:55.352227 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.352207    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-kgv8g\" (UID: \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:55.412243 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.412172    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7srhw\" (UniqueName: \"kubernetes.io/projected/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-kube-api-access-7srhw\") pod \"maas-api-revoke-sub-79805ca9-kgv8g\" (UID: \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:55.458563 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.458238    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:55.476726 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:02:55.476662    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:02:55.850645 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:55.850615    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"]
Oct 31 20:02:55.852030 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:02:55.852002    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1b0db3_aa3f_48d8_aa53_82b36064aae2.slice/crio-2f2b724ea01ca48bff49fe584f8b16b0297b289e5d502ad2db9f640081c2426c WatchSource:0}: Error finding container 2f2b724ea01ca48bff49fe584f8b16b0297b289e5d502ad2db9f640081c2426c: Status 404 returned error can't find the container with id 2f2b724ea01ca48bff49fe584f8b16b0297b289e5d502ad2db9f640081c2426c
Oct 31 20:02:56.420261 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:56.420175    4484 scope.go:122] "RemoveContainer" containerID="639d4f4f9e2d0bf391779a759f5933c88cab4401922b3ee2ddd3107acd08fb95"
Oct 31 20:02:56.568039 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:56.567990    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g" event={"ID":"ab1b0db3-aa3f-48d8-aa53-82b36064aae2","Type":"ContainerStarted","Data":"0f91a46a157b17171791f76f89a4e79b523d7cc1accc614be93d7582892bbd25"}
Oct 31 20:02:56.568236 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:56.568054    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g" event={"ID":"ab1b0db3-aa3f-48d8-aa53-82b36064aae2","Type":"ContainerStarted","Data":"2f2b724ea01ca48bff49fe584f8b16b0297b289e5d502ad2db9f640081c2426c"}
Oct 31 20:02:56.685861 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:56.685751    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g" podStartSLOduration=1.685734901 podStartE2EDuration="1.685734901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:56.684372731 +0000 UTC m=+6122.830822597" watchObservedRunningTime="2026-10-31 20:02:56.685734901 +0000 UTC m=+6122.832184766"
Oct 31 20:02:57.575041 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:57.575000    4484 generic.go:356] "Generic (PLEG): container finished" podID="ab1b0db3-aa3f-48d8-aa53-82b36064aae2" containerID="0f91a46a157b17171791f76f89a4e79b523d7cc1accc614be93d7582892bbd25" exitCode=0
Oct 31 20:02:57.575546 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:57.575089    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g" event={"ID":"ab1b0db3-aa3f-48d8-aa53-82b36064aae2","Type":"ContainerDied","Data":"0f91a46a157b17171791f76f89a4e79b523d7cc1accc614be93d7582892bbd25"}
Oct 31 20:02:58.718771 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:58.718739    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:58.783754 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:58.783722    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-openshift-ca\") pod \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\" (UID: \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\") "
Oct 31 20:02:58.783980 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:58.783834    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-kube-api-access-7srhw\" (UniqueName: \"kubernetes.io/projected/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-kube-api-access-7srhw\") pod \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\" (UID: \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\") "
Oct 31 20:02:58.783980 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:58.783889    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-xks-ca\") pod \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\" (UID: \"ab1b0db3-aa3f-48d8-aa53-82b36064aae2\") "
Oct 31 20:02:58.784103 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:58.784077    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-openshift-ca" pod "ab1b0db3-aa3f-48d8-aa53-82b36064aae2" (UID: "ab1b0db3-aa3f-48d8-aa53-82b36064aae2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:58.784268 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:58.784245    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:02:58.786114 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:58.786090    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-xks-ca" pod "ab1b0db3-aa3f-48d8-aa53-82b36064aae2" (UID: "ab1b0db3-aa3f-48d8-aa53-82b36064aae2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:58.786223 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:58.786134    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-kube-api-access-7srhw" pod "ab1b0db3-aa3f-48d8-aa53-82b36064aae2" (UID: "ab1b0db3-aa3f-48d8-aa53-82b36064aae2"). InnerVolumeSpecName "kube-api-access-7srhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:58.854253 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:58.854170    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:02:58.885047 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:58.885007    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7srhw\" (UniqueName: \"kubernetes.io/projected/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-kube-api-access-7srhw\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:02:58.885047 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:58.885042    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1b0db3-aa3f-48d8-aa53-82b36064aae2-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:02:59.586996 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:59.586956    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"
Oct 31 20:02:59.586996 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:59.586981    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g" event={"ID":"ab1b0db3-aa3f-48d8-aa53-82b36064aae2","Type":"ContainerDied","Data":"2f2b724ea01ca48bff49fe584f8b16b0297b289e5d502ad2db9f640081c2426c"}
Oct 31 20:02:59.587225 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:02:59.587041    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f2b724ea01ca48bff49fe584f8b16b0297b289e5d502ad2db9f640081c2426c"
Oct 31 20:03:07.204384 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:07.204336    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:03:12.582723 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.582678    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"]
Oct 31 20:03:12.583388 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.583362    4484 cpu_manager.go:414] "containerMap: removing container" podUID="ab1b0db3-aa3f-48d8-aa53-82b36064aae2" containerName="revoke-keys"
Oct 31 20:03:12.583525 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.583392    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1b0db3-aa3f-48d8-aa53-82b36064aae2" containerName="revoke-keys"
Oct 31 20:03:12.583605 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.583534    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1b0db3-aa3f-48d8-aa53-82b36064aae2" containerName="revoke-keys"
Oct 31 20:03:12.587578 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.587555    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:12.642516 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.642455    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"]
Oct 31 20:03:12.689770 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.689736    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:03:12.689960 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.689798    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:03:12.715184 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.715147    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vmbx2\" (UID: \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:12.715459 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.715216    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h797s\" (UniqueName: \"kubernetes.io/projected/0ec3bb4e-3275-4f47-a371-87e872f6e7db-kube-api-access-h797s\") pod \"maas-api-revoke-sub-0b7e8edb-vmbx2\" (UID: \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:12.715459 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.715275    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vmbx2\" (UID: \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:12.816802 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.816762    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vmbx2\" (UID: \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:12.817020 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.816848    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h797s\" (UniqueName: \"kubernetes.io/projected/0ec3bb4e-3275-4f47-a371-87e872f6e7db-kube-api-access-h797s\") pod \"maas-api-revoke-sub-0b7e8edb-vmbx2\" (UID: \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:12.817020 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.816901    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vmbx2\" (UID: \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:12.817742 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.817710    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vmbx2\" (UID: \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:12.819719 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.819689    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vmbx2\" (UID: \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:12.875768 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.875698    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h797s\" (UniqueName: \"kubernetes.io/projected/0ec3bb4e-3275-4f47-a371-87e872f6e7db-kube-api-access-h797s\") pod \"maas-api-revoke-sub-0b7e8edb-vmbx2\" (UID: \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:12.902885 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:12.902851    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:13.314543 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:13.314441    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"]
Oct 31 20:03:13.660123 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:13.660089    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2" event={"ID":"0ec3bb4e-3275-4f47-a371-87e872f6e7db","Type":"ContainerStarted","Data":"168a2ba3153c04e5e829c5139b85e4c70d3c3c72a6ec7decd85b2e0037fb30f1"}
Oct 31 20:03:14.696500 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:14.696403    4484 generic.go:356] "Generic (PLEG): container finished" podID="0ec3bb4e-3275-4f47-a371-87e872f6e7db" containerID="9bca35e5e50a8f0a3c0b6359a90e1b41eca747919eebbe8dd763ab18c0ea09fe" exitCode=0
Oct 31 20:03:14.697044 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:14.696509    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2" event={"ID":"0ec3bb4e-3275-4f47-a371-87e872f6e7db","Type":"ContainerDied","Data":"9bca35e5e50a8f0a3c0b6359a90e1b41eca747919eebbe8dd763ab18c0ea09fe"}
Oct 31 20:03:15.270817 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:15.270780    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:03:15.845896 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:15.845867    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:15.860716 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:15.860691    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-xks-ca\") pod \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\" (UID: \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\") "
Oct 31 20:03:15.860885 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:15.860733    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-openshift-ca\") pod \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\" (UID: \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\") "
Oct 31 20:03:15.860885 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:15.860861    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec3bb4e-3275-4f47-a371-87e872f6e7db-kube-api-access-h797s\" (UniqueName: \"kubernetes.io/projected/0ec3bb4e-3275-4f47-a371-87e872f6e7db-kube-api-access-h797s\") pod \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\" (UID: \"0ec3bb4e-3275-4f47-a371-87e872f6e7db\") "
Oct 31 20:03:15.861089 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:15.861070    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-openshift-ca" pod "0ec3bb4e-3275-4f47-a371-87e872f6e7db" (UID: "0ec3bb4e-3275-4f47-a371-87e872f6e7db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:15.861169 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:15.861157    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:03:15.862964 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:15.862940    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-xks-ca" pod "0ec3bb4e-3275-4f47-a371-87e872f6e7db" (UID: "0ec3bb4e-3275-4f47-a371-87e872f6e7db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:15.863089 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:15.863070    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec3bb4e-3275-4f47-a371-87e872f6e7db-kube-api-access-h797s" pod "0ec3bb4e-3275-4f47-a371-87e872f6e7db" (UID: "0ec3bb4e-3275-4f47-a371-87e872f6e7db"). InnerVolumeSpecName "kube-api-access-h797s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:15.962074 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:15.961992    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h797s\" (UniqueName: \"kubernetes.io/projected/0ec3bb4e-3275-4f47-a371-87e872f6e7db-kube-api-access-h797s\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:03:15.962074 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:15.962026    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ec3bb4e-3275-4f47-a371-87e872f6e7db-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:03:16.707967 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:16.707933    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"
Oct 31 20:03:16.708207 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:16.707969    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2" event={"ID":"0ec3bb4e-3275-4f47-a371-87e872f6e7db","Type":"ContainerDied","Data":"168a2ba3153c04e5e829c5139b85e4c70d3c3c72a6ec7decd85b2e0037fb30f1"}
Oct 31 20:03:16.708207 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:16.708019    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="168a2ba3153c04e5e829c5139b85e4c70d3c3c72a6ec7decd85b2e0037fb30f1"
Oct 31 20:03:23.753746 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:23.753712    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:03:23.869501 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:23.869452    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"]
Oct 31 20:03:23.869904 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:23.869891    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0ec3bb4e-3275-4f47-a371-87e872f6e7db" containerName="revoke-keys"
Oct 31 20:03:23.869952 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:23.869904    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec3bb4e-3275-4f47-a371-87e872f6e7db" containerName="revoke-keys"
Oct 31 20:03:23.869996 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:23.869985    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec3bb4e-3275-4f47-a371-87e872f6e7db" containerName="revoke-keys"
Oct 31 20:03:23.873316 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:23.873296    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:23.928348 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:23.928259    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:03:23.931341 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:23.931308    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"]
Oct 31 20:03:23.940761 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:23.940737    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-96cgc\" (UID: \"a6dda4f9-84d4-4b43-be4e-7977083043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:23.940945 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:23.940781    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-96cgc\" (UID: \"a6dda4f9-84d4-4b43-be4e-7977083043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:23.940945 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:23.940804    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dbjn\" (UniqueName: \"kubernetes.io/projected/a6dda4f9-84d4-4b43-be4e-7977083043d2-kube-api-access-8dbjn\") pod \"maas-api-revoke-sub-3d4156f1-96cgc\" (UID: \"a6dda4f9-84d4-4b43-be4e-7977083043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:23.978660 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:23.978626    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:03:24.042307 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:24.042193    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-96cgc\" (UID: \"a6dda4f9-84d4-4b43-be4e-7977083043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:24.042307 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:24.042271    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-96cgc\" (UID: \"a6dda4f9-84d4-4b43-be4e-7977083043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:24.042307 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:24.042301    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dbjn\" (UniqueName: \"kubernetes.io/projected/a6dda4f9-84d4-4b43-be4e-7977083043d2-kube-api-access-8dbjn\") pod \"maas-api-revoke-sub-3d4156f1-96cgc\" (UID: \"a6dda4f9-84d4-4b43-be4e-7977083043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:24.043004 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:24.042983    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-96cgc\" (UID: \"a6dda4f9-84d4-4b43-be4e-7977083043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:24.044868 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:24.044851    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-96cgc\" (UID: \"a6dda4f9-84d4-4b43-be4e-7977083043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:24.101241 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:24.101206    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dbjn\" (UniqueName: \"kubernetes.io/projected/a6dda4f9-84d4-4b43-be4e-7977083043d2-kube-api-access-8dbjn\") pod \"maas-api-revoke-sub-3d4156f1-96cgc\" (UID: \"a6dda4f9-84d4-4b43-be4e-7977083043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:24.193060 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:24.193022    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:24.565755 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:24.565729    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"]
Oct 31 20:03:24.567314 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:03:24.567279    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6dda4f9_84d4_4b43_be4e_7977083043d2.slice/crio-7880d588fae32374c968991e6322d5daea4a152dfbcd1188080cac2c4544755e WatchSource:0}: Error finding container 7880d588fae32374c968991e6322d5daea4a152dfbcd1188080cac2c4544755e: Status 404 returned error can't find the container with id 7880d588fae32374c968991e6322d5daea4a152dfbcd1188080cac2c4544755e
Oct 31 20:03:24.750940 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:24.750902    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc" event={"ID":"a6dda4f9-84d4-4b43-be4e-7977083043d2","Type":"ContainerStarted","Data":"7880d588fae32374c968991e6322d5daea4a152dfbcd1188080cac2c4544755e"}
Oct 31 20:03:25.469873 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:03:25.469833    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:03:25.765783 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:25.758407    4484 generic.go:356] "Generic (PLEG): container finished" podID="a6dda4f9-84d4-4b43-be4e-7977083043d2" containerID="39532a99a6ef9cf77943d3ed48f47522477fc3d2f88f30b89c8155b5eb0d8927" exitCode=0
Oct 31 20:03:25.765783 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:25.758661    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc" event={"ID":"a6dda4f9-84d4-4b43-be4e-7977083043d2","Type":"ContainerDied","Data":"39532a99a6ef9cf77943d3ed48f47522477fc3d2f88f30b89c8155b5eb0d8927"}
Oct 31 20:03:26.910225 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:26.910201    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:26.971427 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:26.971396    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-openshift-ca\") pod \"a6dda4f9-84d4-4b43-be4e-7977083043d2\" (UID: \"a6dda4f9-84d4-4b43-be4e-7977083043d2\") "
Oct 31 20:03:26.971588 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:26.971434    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6dda4f9-84d4-4b43-be4e-7977083043d2-kube-api-access-8dbjn\" (UniqueName: \"kubernetes.io/projected/a6dda4f9-84d4-4b43-be4e-7977083043d2-kube-api-access-8dbjn\") pod \"a6dda4f9-84d4-4b43-be4e-7977083043d2\" (UID: \"a6dda4f9-84d4-4b43-be4e-7977083043d2\") "
Oct 31 20:03:26.971588 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:26.971522    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-xks-ca\") pod \"a6dda4f9-84d4-4b43-be4e-7977083043d2\" (UID: \"a6dda4f9-84d4-4b43-be4e-7977083043d2\") "
Oct 31 20:03:26.971774 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:26.971745    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-openshift-ca" pod "a6dda4f9-84d4-4b43-be4e-7977083043d2" (UID: "a6dda4f9-84d4-4b43-be4e-7977083043d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:26.973672 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:26.973642    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6dda4f9-84d4-4b43-be4e-7977083043d2-kube-api-access-8dbjn" pod "a6dda4f9-84d4-4b43-be4e-7977083043d2" (UID: "a6dda4f9-84d4-4b43-be4e-7977083043d2"). InnerVolumeSpecName "kube-api-access-8dbjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:26.973850 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:26.973831    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-xks-ca" pod "a6dda4f9-84d4-4b43-be4e-7977083043d2" (UID: "a6dda4f9-84d4-4b43-be4e-7977083043d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:27.072254 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:27.072186    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:03:27.072254 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:27.072210    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6dda4f9-84d4-4b43-be4e-7977083043d2-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:03:27.072254 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:27.072222    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dbjn\" (UniqueName: \"kubernetes.io/projected/a6dda4f9-84d4-4b43-be4e-7977083043d2-kube-api-access-8dbjn\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:03:27.770913 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:27.770879    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"
Oct 31 20:03:27.771094 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:27.770873    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc" event={"ID":"a6dda4f9-84d4-4b43-be4e-7977083043d2","Type":"ContainerDied","Data":"7880d588fae32374c968991e6322d5daea4a152dfbcd1188080cac2c4544755e"}
Oct 31 20:03:27.771094 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:27.771015    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7880d588fae32374c968991e6322d5daea4a152dfbcd1188080cac2c4544755e"
Oct 31 20:03:38.469991 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:38.469911    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:03:47.453693 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:47.453644    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-g4xdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:03:47.493499 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:47.489107    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" podUID="a7dcf339-7a9f-47df-acef-a2ab94742e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:03:48.453512 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.453451    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"]
Oct 31 20:03:48.504491 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.504445    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-96cgc"]
Oct 31 20:03:48.577992 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.577949    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"]
Oct 31 20:03:48.578400 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.578385    4484 cpu_manager.go:414] "containerMap: removing container" podUID="a6dda4f9-84d4-4b43-be4e-7977083043d2" containerName="revoke-keys"
Oct 31 20:03:48.578400 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.578399    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6dda4f9-84d4-4b43-be4e-7977083043d2" containerName="revoke-keys"
Oct 31 20:03:48.578555 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.578494    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6dda4f9-84d4-4b43-be4e-7977083043d2" containerName="revoke-keys"
Oct 31 20:03:48.581544 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.581524    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:48.636567 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.636522    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:03:48.636567 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.636537    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:03:48.639416 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.639382    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"]
Oct 31 20:03:48.650121 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.650089    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-498x8\" (UID: \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:48.650347 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.650203    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-498x8\" (UID: \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:48.650347 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.650250    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd67l\" (UniqueName: \"kubernetes.io/projected/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-kube-api-access-xd67l\") pod \"maas-api-revoke-sub-3d4156f1-498x8\" (UID: \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:48.751097 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.751056    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-498x8\" (UID: \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:48.751097 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.751104    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd67l\" (UniqueName: \"kubernetes.io/projected/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-kube-api-access-xd67l\") pod \"maas-api-revoke-sub-3d4156f1-498x8\" (UID: \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:48.751382 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.751157    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-498x8\" (UID: \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:48.751867 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.751840    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-498x8\" (UID: \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:48.753862 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.753841    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-498x8\" (UID: \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:48.809836 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.809801    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd67l\" (UniqueName: \"kubernetes.io/projected/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-kube-api-access-xd67l\") pod \"maas-api-revoke-sub-3d4156f1-498x8\" (UID: \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:48.893872 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:48.893836    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:49.269416 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:49.269380    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"]
Oct 31 20:03:49.270903 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:03:49.270875    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44baab7_1ec1_4578_813b_8c03b0c7ab5d.slice/crio-a3f64fe317f88b586bf14db5326ab5c4ab5d365da4594707ff0f3e36677b2d02 WatchSource:0}: Error finding container a3f64fe317f88b586bf14db5326ab5c4ab5d365da4594707ff0f3e36677b2d02: Status 404 returned error can't find the container with id a3f64fe317f88b586bf14db5326ab5c4ab5d365da4594707ff0f3e36677b2d02
Oct 31 20:03:49.924628 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:49.924530    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8" event={"ID":"f44baab7-1ec1-4578-813b-8c03b0c7ab5d","Type":"ContainerStarted","Data":"d53c1b6ed2a8dc2b573b2b5f4b28b41c821cd946faa68a745b3be26c046fabaf"}
Oct 31 20:03:49.924628 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:49.924596    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8" event={"ID":"f44baab7-1ec1-4578-813b-8c03b0c7ab5d","Type":"ContainerStarted","Data":"a3f64fe317f88b586bf14db5326ab5c4ab5d365da4594707ff0f3e36677b2d02"}
Oct 31 20:03:49.963701 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:49.963665    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:03:49.992619 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:49.992588    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6dda4f9-84d4-4b43-be4e-7977083043d2" path="/var/lib/kubelet/pods/a6dda4f9-84d4-4b43-be4e-7977083043d2/volumes"
Oct 31 20:03:50.076859 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:50.076803    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8" podStartSLOduration=2.076787025 podStartE2EDuration="2.076787025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:50.074815537 +0000 UTC m=+6176.221265407" watchObservedRunningTime="2026-10-31 20:03:50.076787025 +0000 UTC m=+6176.223236910"
Oct 31 20:03:50.931012 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:50.930975    4484 generic.go:356] "Generic (PLEG): container finished" podID="f44baab7-1ec1-4578-813b-8c03b0c7ab5d" containerID="d53c1b6ed2a8dc2b573b2b5f4b28b41c821cd946faa68a745b3be26c046fabaf" exitCode=0
Oct 31 20:03:50.931012 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:50.931016    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8" event={"ID":"f44baab7-1ec1-4578-813b-8c03b0c7ab5d","Type":"ContainerDied","Data":"d53c1b6ed2a8dc2b573b2b5f4b28b41c821cd946faa68a745b3be26c046fabaf"}
Oct 31 20:03:52.085809 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.085771    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:52.177738 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.177708    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-openshift-ca\") pod \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\" (UID: \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\") "
Oct 31 20:03:52.177928 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.177770    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-kube-api-access-xd67l\" (UniqueName: \"kubernetes.io/projected/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-kube-api-access-xd67l\") pod \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\" (UID: \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\") "
Oct 31 20:03:52.178069 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.178048    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-openshift-ca" pod "f44baab7-1ec1-4578-813b-8c03b0c7ab5d" (UID: "f44baab7-1ec1-4578-813b-8c03b0c7ab5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:52.180064 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.180041    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-kube-api-access-xd67l" pod "f44baab7-1ec1-4578-813b-8c03b0c7ab5d" (UID: "f44baab7-1ec1-4578-813b-8c03b0c7ab5d"). InnerVolumeSpecName "kube-api-access-xd67l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:52.278505 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.278436    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-xks-ca\") pod \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\" (UID: \"f44baab7-1ec1-4578-813b-8c03b0c7ab5d\") "
Oct 31 20:03:52.278722 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.278677    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:03:52.278722 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.278691    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd67l\" (UniqueName: \"kubernetes.io/projected/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-kube-api-access-xd67l\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:03:52.280800 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.280772    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-xks-ca" pod "f44baab7-1ec1-4578-813b-8c03b0c7ab5d" (UID: "f44baab7-1ec1-4578-813b-8c03b0c7ab5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:52.379169 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.379128    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f44baab7-1ec1-4578-813b-8c03b0c7ab5d-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:03:52.942151 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.942117    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8" event={"ID":"f44baab7-1ec1-4578-813b-8c03b0c7ab5d","Type":"ContainerDied","Data":"a3f64fe317f88b586bf14db5326ab5c4ab5d365da4594707ff0f3e36677b2d02"}
Oct 31 20:03:52.942151 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.942140    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"
Oct 31 20:03:52.942435 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:52.942166    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3f64fe317f88b586bf14db5326ab5c4ab5d365da4594707ff0f3e36677b2d02"
Oct 31 20:03:55.469639 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:03:55.469601    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:03:56.442043 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:03:56.442005    4484 scope.go:122] "RemoveContainer" containerID="27adc882ea591e1a9041449a09e14a9e5f7ce70c5cc26360c64e6f37a1d6b615"
Oct 31 20:04:03.166729 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:03.166691    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:04:13.093785 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:13.093750    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:04:24.851465 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:24.851420    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:04:25.469463 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:04:25.469429    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:04:33.736203 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:33.736169    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"]
Oct 31 20:04:33.736648 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:33.736635    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f44baab7-1ec1-4578-813b-8c03b0c7ab5d" containerName="revoke-keys"
Oct 31 20:04:33.736698 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:33.736649    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44baab7-1ec1-4578-813b-8c03b0c7ab5d" containerName="revoke-keys"
Oct 31 20:04:33.736756 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:33.736747    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44baab7-1ec1-4578-813b-8c03b0c7ab5d" containerName="revoke-keys"
Oct 31 20:04:33.740128 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:33.740104    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:33.796906 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:33.796850    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"]
Oct 31 20:04:33.844420 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:33.844387    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:04:33.844615 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:33.844521    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:04:33.911263 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:33.911215    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gpkn\" (UniqueName: \"kubernetes.io/projected/e2fecf6d-0090-4a15-971b-729227ce1de2-kube-api-access-2gpkn\") pod \"maas-api-revoke-sub-e2fc9c5b-5mmz2\" (UID: \"e2fecf6d-0090-4a15-971b-729227ce1de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:33.911263 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:33.911260    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5mmz2\" (UID: \"e2fecf6d-0090-4a15-971b-729227ce1de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:33.911614 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:33.911284    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5mmz2\" (UID: \"e2fecf6d-0090-4a15-971b-729227ce1de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:34.012827 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:34.012749    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5mmz2\" (UID: \"e2fecf6d-0090-4a15-971b-729227ce1de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:34.012827 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:34.012785    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5mmz2\" (UID: \"e2fecf6d-0090-4a15-971b-729227ce1de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:34.013020 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:34.012875    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gpkn\" (UniqueName: \"kubernetes.io/projected/e2fecf6d-0090-4a15-971b-729227ce1de2-kube-api-access-2gpkn\") pod \"maas-api-revoke-sub-e2fc9c5b-5mmz2\" (UID: \"e2fecf6d-0090-4a15-971b-729227ce1de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:34.013638 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:34.013566    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5mmz2\" (UID: \"e2fecf6d-0090-4a15-971b-729227ce1de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:34.015361 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:34.015331    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-5mmz2\" (UID: \"e2fecf6d-0090-4a15-971b-729227ce1de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:34.071424 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:34.071396    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gpkn\" (UniqueName: \"kubernetes.io/projected/e2fecf6d-0090-4a15-971b-729227ce1de2-kube-api-access-2gpkn\") pod \"maas-api-revoke-sub-e2fc9c5b-5mmz2\" (UID: \"e2fecf6d-0090-4a15-971b-729227ce1de2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:34.353043 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:34.352952    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:34.740365 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:34.740341    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"]
Oct 31 20:04:34.741330 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:04:34.741292    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fecf6d_0090_4a15_971b_729227ce1de2.slice/crio-81ae1107c5f822264f6b44558966387959ccbf7722a4f1531ad707f80af90694 WatchSource:0}: Error finding container 81ae1107c5f822264f6b44558966387959ccbf7722a4f1531ad707f80af90694: Status 404 returned error can't find the container with id 81ae1107c5f822264f6b44558966387959ccbf7722a4f1531ad707f80af90694
Oct 31 20:04:35.193220 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:35.193184    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2" event={"ID":"e2fecf6d-0090-4a15-971b-729227ce1de2","Type":"ContainerStarted","Data":"01770e0e5543f47ec4959ec72a63552d914d256999589bb1faf0e883b6eaaa28"}
Oct 31 20:04:35.193447 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:35.193247    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2" event={"ID":"e2fecf6d-0090-4a15-971b-729227ce1de2","Type":"ContainerStarted","Data":"81ae1107c5f822264f6b44558966387959ccbf7722a4f1531ad707f80af90694"}
Oct 31 20:04:35.310294 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:35.310240    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2" podStartSLOduration=2.310226022 podStartE2EDuration="2.310226022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:35.309664344 +0000 UTC m=+6221.456114208" watchObservedRunningTime="2026-10-31 20:04:35.310226022 +0000 UTC m=+6221.456675878"
Oct 31 20:04:36.201158 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:36.201120    4484 generic.go:356] "Generic (PLEG): container finished" podID="e2fecf6d-0090-4a15-971b-729227ce1de2" containerID="01770e0e5543f47ec4959ec72a63552d914d256999589bb1faf0e883b6eaaa28" exitCode=0
Oct 31 20:04:36.201619 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:36.201204    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2" event={"ID":"e2fecf6d-0090-4a15-971b-729227ce1de2","Type":"ContainerDied","Data":"01770e0e5543f47ec4959ec72a63552d914d256999589bb1faf0e883b6eaaa28"}
Oct 31 20:04:37.347383 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:37.347358    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:37.450205 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:37.450167    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-openshift-ca\") pod \"e2fecf6d-0090-4a15-971b-729227ce1de2\" (UID: \"e2fecf6d-0090-4a15-971b-729227ce1de2\") "
Oct 31 20:04:37.450401 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:37.450246    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2fecf6d-0090-4a15-971b-729227ce1de2-kube-api-access-2gpkn\" (UniqueName: \"kubernetes.io/projected/e2fecf6d-0090-4a15-971b-729227ce1de2-kube-api-access-2gpkn\") pod \"e2fecf6d-0090-4a15-971b-729227ce1de2\" (UID: \"e2fecf6d-0090-4a15-971b-729227ce1de2\") "
Oct 31 20:04:37.450401 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:37.450278    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-xks-ca\") pod \"e2fecf6d-0090-4a15-971b-729227ce1de2\" (UID: \"e2fecf6d-0090-4a15-971b-729227ce1de2\") "
Oct 31 20:04:37.450595 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:37.450570    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-openshift-ca" pod "e2fecf6d-0090-4a15-971b-729227ce1de2" (UID: "e2fecf6d-0090-4a15-971b-729227ce1de2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:37.452623 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:37.452598    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-xks-ca" pod "e2fecf6d-0090-4a15-971b-729227ce1de2" (UID: "e2fecf6d-0090-4a15-971b-729227ce1de2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:37.452717 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:37.452625    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fecf6d-0090-4a15-971b-729227ce1de2-kube-api-access-2gpkn" pod "e2fecf6d-0090-4a15-971b-729227ce1de2" (UID: "e2fecf6d-0090-4a15-971b-729227ce1de2"). InnerVolumeSpecName "kube-api-access-2gpkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:37.551607 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:37.551570    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:04:37.551607 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:37.551602    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gpkn\" (UniqueName: \"kubernetes.io/projected/e2fecf6d-0090-4a15-971b-729227ce1de2-kube-api-access-2gpkn\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:04:37.551607 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:37.551611    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2fecf6d-0090-4a15-971b-729227ce1de2-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:04:38.213459 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:38.213421    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"
Oct 31 20:04:38.213650 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:38.213424    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2" event={"ID":"e2fecf6d-0090-4a15-971b-729227ce1de2","Type":"ContainerDied","Data":"81ae1107c5f822264f6b44558966387959ccbf7722a4f1531ad707f80af90694"}
Oct 31 20:04:38.213650 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:38.213528    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81ae1107c5f822264f6b44558966387959ccbf7722a4f1531ad707f80af90694"
Oct 31 20:04:44.488253 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.488222    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"]
Oct 31 20:04:44.488792 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.488705    4484 cpu_manager.go:414] "containerMap: removing container" podUID="e2fecf6d-0090-4a15-971b-729227ce1de2" containerName="revoke-keys"
Oct 31 20:04:44.488792 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.488718    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2fecf6d-0090-4a15-971b-729227ce1de2" containerName="revoke-keys"
Oct 31 20:04:44.488792 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.488780    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2fecf6d-0090-4a15-971b-729227ce1de2" containerName="revoke-keys"
Oct 31 20:04:44.492146 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.492121    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:44.516538 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.516506    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-vz6rq\" (UID: \"7a54f96a-7f15-4b54-a81c-f721561c8def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:44.516688 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.516569    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-vz6rq\" (UID: \"7a54f96a-7f15-4b54-a81c-f721561c8def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:44.516688 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.516644    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz6jr\" (UniqueName: \"kubernetes.io/projected/7a54f96a-7f15-4b54-a81c-f721561c8def-kube-api-access-sz6jr\") pod \"maas-api-revoke-sub-5a29efb3-vz6rq\" (UID: \"7a54f96a-7f15-4b54-a81c-f721561c8def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:44.549503 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.549446    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"]
Oct 31 20:04:44.596316 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.596283    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:04:44.596533 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.596337    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:04:44.617843 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.617801    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-vz6rq\" (UID: \"7a54f96a-7f15-4b54-a81c-f721561c8def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:44.618050 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.617867    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-vz6rq\" (UID: \"7a54f96a-7f15-4b54-a81c-f721561c8def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:44.618050 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.617910    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6jr\" (UniqueName: \"kubernetes.io/projected/7a54f96a-7f15-4b54-a81c-f721561c8def-kube-api-access-sz6jr\") pod \"maas-api-revoke-sub-5a29efb3-vz6rq\" (UID: \"7a54f96a-7f15-4b54-a81c-f721561c8def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:44.618618 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.618550    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-vz6rq\" (UID: \"7a54f96a-7f15-4b54-a81c-f721561c8def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:44.620268 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.620241    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-vz6rq\" (UID: \"7a54f96a-7f15-4b54-a81c-f721561c8def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:44.676182 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.676144    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6jr\" (UniqueName: \"kubernetes.io/projected/7a54f96a-7f15-4b54-a81c-f721561c8def-kube-api-access-sz6jr\") pod \"maas-api-revoke-sub-5a29efb3-vz6rq\" (UID: \"7a54f96a-7f15-4b54-a81c-f721561c8def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:44.806382 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:44.806292    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:45.178269 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:45.178246    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"]
Oct 31 20:04:45.252728 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:45.252692    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq" event={"ID":"7a54f96a-7f15-4b54-a81c-f721561c8def","Type":"ContainerStarted","Data":"de0e3dd8c9354d0950462749e040b50cb1cbd89a81890ebc4585bda4a18d1343"}
Oct 31 20:04:46.259608 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:46.259573    4484 generic.go:356] "Generic (PLEG): container finished" podID="7a54f96a-7f15-4b54-a81c-f721561c8def" containerID="8f3ca92a77ddb8f753a629d9b199495c498d6b3bbdf54edce6f7f7552a7c7b0f" exitCode=0
Oct 31 20:04:46.260074 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:46.259664    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq" event={"ID":"7a54f96a-7f15-4b54-a81c-f721561c8def","Type":"ContainerDied","Data":"8f3ca92a77ddb8f753a629d9b199495c498d6b3bbdf54edce6f7f7552a7c7b0f"}
Oct 31 20:04:46.569877 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:46.569831    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:04:47.442535 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:47.442505    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:47.549098 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:47.548986    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-openshift-ca\") pod \"7a54f96a-7f15-4b54-a81c-f721561c8def\" (UID: \"7a54f96a-7f15-4b54-a81c-f721561c8def\") "
Oct 31 20:04:47.549098 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:47.549055    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a54f96a-7f15-4b54-a81c-f721561c8def-kube-api-access-sz6jr\" (UniqueName: \"kubernetes.io/projected/7a54f96a-7f15-4b54-a81c-f721561c8def-kube-api-access-sz6jr\") pod \"7a54f96a-7f15-4b54-a81c-f721561c8def\" (UID: \"7a54f96a-7f15-4b54-a81c-f721561c8def\") "
Oct 31 20:04:47.549098 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:47.549090    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-xks-ca\") pod \"7a54f96a-7f15-4b54-a81c-f721561c8def\" (UID: \"7a54f96a-7f15-4b54-a81c-f721561c8def\") "
Oct 31 20:04:47.549988 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:47.549943    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-openshift-ca" pod "7a54f96a-7f15-4b54-a81c-f721561c8def" (UID: "7a54f96a-7f15-4b54-a81c-f721561c8def"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:47.552404 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:47.552288    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a54f96a-7f15-4b54-a81c-f721561c8def-kube-api-access-sz6jr" pod "7a54f96a-7f15-4b54-a81c-f721561c8def" (UID: "7a54f96a-7f15-4b54-a81c-f721561c8def"). InnerVolumeSpecName "kube-api-access-sz6jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:47.552404 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:47.552368    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-xks-ca" pod "7a54f96a-7f15-4b54-a81c-f721561c8def" (UID: "7a54f96a-7f15-4b54-a81c-f721561c8def"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:47.650977 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:47.650927    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:04:47.650977 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:47.650970    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz6jr\" (UniqueName: \"kubernetes.io/projected/7a54f96a-7f15-4b54-a81c-f721561c8def-kube-api-access-sz6jr\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:04:47.650977 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:47.650987    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a54f96a-7f15-4b54-a81c-f721561c8def-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:04:48.288912 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:48.288859    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq" event={"ID":"7a54f96a-7f15-4b54-a81c-f721561c8def","Type":"ContainerDied","Data":"de0e3dd8c9354d0950462749e040b50cb1cbd89a81890ebc4585bda4a18d1343"}
Oct 31 20:04:48.289224 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:48.289188    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de0e3dd8c9354d0950462749e040b50cb1cbd89a81890ebc4585bda4a18d1343"
Oct 31 20:04:48.289500 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:48.289451    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"
Oct 31 20:04:48.948662 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:48.948632    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vnvmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:04:48.949096 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:48.948684    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podUID="a9f80747-2277-4637-a04a-15207891a808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:04:55.469791 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:04:55.469755    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:04:57.256210 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:04:57.256169    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:05:01.560694 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.560655    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"]
Oct 31 20:05:01.611690 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.611652    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-498x8"]
Oct 31 20:05:01.683764 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.683728    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"]
Oct 31 20:05:01.684171 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.684159    4484 cpu_manager.go:414] "containerMap: removing container" podUID="7a54f96a-7f15-4b54-a81c-f721561c8def" containerName="revoke-keys"
Oct 31 20:05:01.684218 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.684173    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a54f96a-7f15-4b54-a81c-f721561c8def" containerName="revoke-keys"
Oct 31 20:05:01.684252 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.684247    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a54f96a-7f15-4b54-a81c-f721561c8def" containerName="revoke-keys"
Oct 31 20:05:01.687748 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.687730    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:01.743233 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.743196    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:05:01.743438 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.743316    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:05:01.746790 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.746757    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"]
Oct 31 20:05:01.824593 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.824503    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v99nw\" (UID: \"0f59c66e-9739-4d5f-a848-65b4b350f640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:01.824752 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.824665    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcs6n\" (UniqueName: \"kubernetes.io/projected/0f59c66e-9739-4d5f-a848-65b4b350f640-kube-api-access-lcs6n\") pod \"maas-api-revoke-sub-3d4156f1-v99nw\" (UID: \"0f59c66e-9739-4d5f-a848-65b4b350f640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:01.824752 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.824710    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v99nw\" (UID: \"0f59c66e-9739-4d5f-a848-65b4b350f640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:01.925927 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.925882    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v99nw\" (UID: \"0f59c66e-9739-4d5f-a848-65b4b350f640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:01.926122 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.925942    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v99nw\" (UID: \"0f59c66e-9739-4d5f-a848-65b4b350f640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:01.926122 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.926016    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcs6n\" (UniqueName: \"kubernetes.io/projected/0f59c66e-9739-4d5f-a848-65b4b350f640-kube-api-access-lcs6n\") pod \"maas-api-revoke-sub-3d4156f1-v99nw\" (UID: \"0f59c66e-9739-4d5f-a848-65b4b350f640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:01.926742 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.926712    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v99nw\" (UID: \"0f59c66e-9739-4d5f-a848-65b4b350f640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:01.928661 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.928641    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v99nw\" (UID: \"0f59c66e-9739-4d5f-a848-65b4b350f640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:01.984993 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.984958    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcs6n\" (UniqueName: \"kubernetes.io/projected/0f59c66e-9739-4d5f-a848-65b4b350f640-kube-api-access-lcs6n\") pod \"maas-api-revoke-sub-3d4156f1-v99nw\" (UID: \"0f59c66e-9739-4d5f-a848-65b4b350f640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:01.994255 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:01.994217    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f44baab7-1ec1-4578-813b-8c03b0c7ab5d" path="/var/lib/kubelet/pods/f44baab7-1ec1-4578-813b-8c03b0c7ab5d/volumes"
Oct 31 20:05:02.001080 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:02.001056    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:02.376592 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:02.376564    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"]
Oct 31 20:05:02.378427 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:05:02.378399    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f59c66e_9739_4d5f_a848_65b4b350f640.slice/crio-bc0b2c34f888457733a38fa8078cd27229d279f4411e4c3df865ba451b7e007b WatchSource:0}: Error finding container bc0b2c34f888457733a38fa8078cd27229d279f4411e4c3df865ba451b7e007b: Status 404 returned error can't find the container with id bc0b2c34f888457733a38fa8078cd27229d279f4411e4c3df865ba451b7e007b
Oct 31 20:05:03.390183 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:03.390151    4484 generic.go:356] "Generic (PLEG): container finished" podID="0f59c66e-9739-4d5f-a848-65b4b350f640" containerID="76a91c28cc7fef243d3dab42787b736ab8b1a3eb395d3abbfe191776e2f69d58" exitCode=0
Oct 31 20:05:03.390630 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:03.390231    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw" event={"ID":"0f59c66e-9739-4d5f-a848-65b4b350f640","Type":"ContainerDied","Data":"76a91c28cc7fef243d3dab42787b736ab8b1a3eb395d3abbfe191776e2f69d58"}
Oct 31 20:05:03.390630 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:03.390280    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw" event={"ID":"0f59c66e-9739-4d5f-a848-65b4b350f640","Type":"ContainerStarted","Data":"bc0b2c34f888457733a38fa8078cd27229d279f4411e4c3df865ba451b7e007b"}
Oct 31 20:05:04.538664 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:04.538628    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:04.651894 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:04.651805    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-openshift-ca\") pod \"0f59c66e-9739-4d5f-a848-65b4b350f640\" (UID: \"0f59c66e-9739-4d5f-a848-65b4b350f640\") "
Oct 31 20:05:04.651894 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:04.651876    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f59c66e-9739-4d5f-a848-65b4b350f640-kube-api-access-lcs6n\" (UniqueName: \"kubernetes.io/projected/0f59c66e-9739-4d5f-a848-65b4b350f640-kube-api-access-lcs6n\") pod \"0f59c66e-9739-4d5f-a848-65b4b350f640\" (UID: \"0f59c66e-9739-4d5f-a848-65b4b350f640\") "
Oct 31 20:05:04.652108 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:04.651905    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-xks-ca\") pod \"0f59c66e-9739-4d5f-a848-65b4b350f640\" (UID: \"0f59c66e-9739-4d5f-a848-65b4b350f640\") "
Oct 31 20:05:04.652177 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:04.652156    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-openshift-ca" pod "0f59c66e-9739-4d5f-a848-65b4b350f640" (UID: "0f59c66e-9739-4d5f-a848-65b4b350f640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:04.654067 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:04.654040    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-xks-ca" pod "0f59c66e-9739-4d5f-a848-65b4b350f640" (UID: "0f59c66e-9739-4d5f-a848-65b4b350f640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:04.654207 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:04.654082    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f59c66e-9739-4d5f-a848-65b4b350f640-kube-api-access-lcs6n" pod "0f59c66e-9739-4d5f-a848-65b4b350f640" (UID: "0f59c66e-9739-4d5f-a848-65b4b350f640"). InnerVolumeSpecName "kube-api-access-lcs6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:04.752870 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:04.752831    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:05:04.752870 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:04.752867    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcs6n\" (UniqueName: \"kubernetes.io/projected/0f59c66e-9739-4d5f-a848-65b4b350f640-kube-api-access-lcs6n\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:05:04.752870 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:04.752877    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f59c66e-9739-4d5f-a848-65b4b350f640-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:05:05.402247 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:05.402218    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"
Oct 31 20:05:05.402437 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:05.402253    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw" event={"ID":"0f59c66e-9739-4d5f-a848-65b4b350f640","Type":"ContainerDied","Data":"bc0b2c34f888457733a38fa8078cd27229d279f4411e4c3df865ba451b7e007b"}
Oct 31 20:05:05.402437 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:05.402302    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc0b2c34f888457733a38fa8078cd27229d279f4411e4c3df865ba451b7e007b"
Oct 31 20:05:16.348447 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:16.348407    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:05:18.341058 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.341018    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"]
Oct 31 20:05:18.342323 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.342293    4484 cpu_manager.go:414] "containerMap: removing container" podUID="0f59c66e-9739-4d5f-a848-65b4b350f640" containerName="revoke-keys"
Oct 31 20:05:18.342510 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.342491    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f59c66e-9739-4d5f-a848-65b4b350f640" containerName="revoke-keys"
Oct 31 20:05:18.342756 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.342739    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f59c66e-9739-4d5f-a848-65b4b350f640" containerName="revoke-keys"
Oct 31 20:05:18.346535 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.346520    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.435539 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.435468    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jwzh\" (UniqueName: \"kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-kube-api-access-2jwzh\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.435822 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.435802    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-credential-socket\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.435956 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.435940    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-envoy\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.436101 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.436084    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd18916-06fc-454a-afd2-c6a736b72331-istio-podinfo\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.436253 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.436234    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd18916-06fc-454a-afd2-c6a736b72331-istiod-ca-cert\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.436393 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.436377    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-socket\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.436595 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.436575    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-istio-token\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.436748 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.436729    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-certs\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.436897 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.436877    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-data\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.437402 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.437380    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-42030c-openshift-default-dockercfg-kvvlm\""
Oct 31 20:05:18.460683 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.460631    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"]
Oct 31 20:05:18.539830 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.537993    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jwzh\" (UniqueName: \"kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-kube-api-access-2jwzh\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.539830 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.538054    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-credential-socket\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.539830 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.538083    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-envoy\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.539830 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.538125    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd18916-06fc-454a-afd2-c6a736b72331-istio-podinfo\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.539830 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.538175    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd18916-06fc-454a-afd2-c6a736b72331-istiod-ca-cert\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.539830 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.538216    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-socket\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.539830 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.538292    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-istio-token\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.539830 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.538326    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-certs\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.539830 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.538365    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-data\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.539830 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.538830    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-data\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.539830 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.539429    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-credential-socket\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.544232 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.543828    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-socket\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.546396 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.546005    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-certs\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.548703 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.547676    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd18916-06fc-454a-afd2-c6a736b72331-istiod-ca-cert\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.548822 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.548718    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd18916-06fc-454a-afd2-c6a736b72331-istio-podinfo\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.549719 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.549671    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-envoy\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.597845 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.597772    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jwzh\" (UniqueName: \"kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-kube-api-access-2jwzh\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.604208 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.604178    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-istio-token\") pod \"e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") " pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:18.660927 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:18.660890    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:19.085965 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:19.085933    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"]
Oct 31 20:05:19.086602 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:05:19.086575    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd18916_06fc_454a_afd2_c6a736b72331.slice/crio-4d7b9b3753008a140a7e5e3423a4a82512d334b79c2179018b6d51ac05348327 WatchSource:0}: Error finding container 4d7b9b3753008a140a7e5e3423a4a82512d334b79c2179018b6d51ac05348327: Status 404 returned error can't find the container with id 4d7b9b3753008a140a7e5e3423a4a82512d334b79c2179018b6d51ac05348327
Oct 31 20:05:19.088837 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:19.088807    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 20:05:19.088893 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:19.088871    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 20:05:19.088930 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:19.088911    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 20:05:19.493271 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:19.493241    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" event={"ID":"2fd18916-06fc-454a-afd2-c6a736b72331","Type":"ContainerStarted","Data":"415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a"}
Oct 31 20:05:19.493704 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:19.493305    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" event={"ID":"2fd18916-06fc-454a-afd2-c6a736b72331","Type":"ContainerStarted","Data":"4d7b9b3753008a140a7e5e3423a4a82512d334b79c2179018b6d51ac05348327"}
Oct 31 20:05:19.617116 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:19.617057    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" podStartSLOduration=1.617041899 podStartE2EDuration="1.617041899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:19.61443852 +0000 UTC m=+6265.760888386" watchObservedRunningTime="2026-10-31 20:05:19.617041899 +0000 UTC m=+6265.763491764"
Oct 31 20:05:19.661935 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:19.661908    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:20.666010 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:20.665980    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:21.503836 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:21.503807    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:21.504985 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:21.504964    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:25.470063 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:05:25.470025    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:05:26.223707 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.223670    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"]
Oct 31 20:05:26.227429 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.227409    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"
Oct 31 20:05:26.285051 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.285021    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"]
Oct 31 20:05:26.312392 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.312355    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc\" (UID: \"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"
Oct 31 20:05:26.312696 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.312675    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxds\" (UniqueName: \"kubernetes.io/projected/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-kube-api-access-dlxds\") pod \"payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc\" (UID: \"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"
Oct 31 20:05:26.332534 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.332500    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-d8444b7b-dockercfg-x2mwd\""
Oct 31 20:05:26.332534 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.332527    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-42030c\""
Oct 31 20:05:26.377213 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.377179    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"]
Oct 31 20:05:26.380876 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.380850    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"
Oct 31 20:05:26.413923 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.413889    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc\" (UID: \"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"
Oct 31 20:05:26.414097 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.413930    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxds\" (UniqueName: \"kubernetes.io/projected/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-kube-api-access-dlxds\") pod \"payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc\" (UID: \"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"
Oct 31 20:05:26.414097 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.413977    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29fqm\" (UniqueName: \"kubernetes.io/projected/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-kube-api-access-29fqm\") pod \"payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx\" (UID: \"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"
Oct 31 20:05:26.414097 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.414079    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx\" (UID: \"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"
Oct 31 20:05:26.414643 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.414623    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc\" (UID: \"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"
Oct 31 20:05:26.439148 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.439116    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"]
Oct 31 20:05:26.475256 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.475183    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxds\" (UniqueName: \"kubernetes.io/projected/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-kube-api-access-dlxds\") pod \"payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc\" (UID: \"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"
Oct 31 20:05:26.514824 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.514785    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29fqm\" (UniqueName: \"kubernetes.io/projected/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-kube-api-access-29fqm\") pod \"payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx\" (UID: \"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"
Oct 31 20:05:26.514985 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.514904    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx\" (UID: \"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"
Oct 31 20:05:26.515612 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.515592    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx\" (UID: \"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"
Oct 31 20:05:26.541359 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.541331    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"
Oct 31 20:05:26.573988 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.573958    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29fqm\" (UniqueName: \"kubernetes.io/projected/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-kube-api-access-29fqm\") pod \"payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx\" (UID: \"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"
Oct 31 20:05:26.694620 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.694584    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"
Oct 31 20:05:26.918857 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:26.918826    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"]
Oct 31 20:05:26.920144 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:05:26.920118    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca8d0f5_3040_4ab5_aaa3_ab6383332fdc.slice/crio-97908e18f23c190c04b27a943408b1b015d564a7e5b42dd91068c4368e87c37e WatchSource:0}: Error finding container 97908e18f23c190c04b27a943408b1b015d564a7e5b42dd91068c4368e87c37e: Status 404 returned error can't find the container with id 97908e18f23c190c04b27a943408b1b015d564a7e5b42dd91068c4368e87c37e
Oct 31 20:05:27.079867 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:27.079838    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"]
Oct 31 20:05:27.081587 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:05:27.081558    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c6f20b_c1fd_49db_bfb2_c5d0fda2bcee.slice/crio-ff217d2d31bad07b5fcb877610b8bd7d76c3a353183b71877dd5a17aee85a4f0 WatchSource:0}: Error finding container ff217d2d31bad07b5fcb877610b8bd7d76c3a353183b71877dd5a17aee85a4f0: Status 404 returned error can't find the container with id ff217d2d31bad07b5fcb877610b8bd7d76c3a353183b71877dd5a17aee85a4f0
Oct 31 20:05:27.194245 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:27.194212    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:05:27.537926 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:27.537888    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc" event={"ID":"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc","Type":"ContainerStarted","Data":"9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc"}
Oct 31 20:05:27.538384 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:27.537958    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"
Oct 31 20:05:27.538384 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:27.537982    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc" event={"ID":"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc","Type":"ContainerStarted","Data":"97908e18f23c190c04b27a943408b1b015d564a7e5b42dd91068c4368e87c37e"}
Oct 31 20:05:27.539509 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:27.539488    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx" event={"ID":"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee","Type":"ContainerStarted","Data":"804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9"}
Oct 31 20:05:27.539570 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:27.539532    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx" event={"ID":"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee","Type":"ContainerStarted","Data":"ff217d2d31bad07b5fcb877610b8bd7d76c3a353183b71877dd5a17aee85a4f0"}
Oct 31 20:05:27.539624 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:27.539609    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"
Oct 31 20:05:27.656230 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:27.656177    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc" podStartSLOduration=1.656163076 podStartE2EDuration="1.656163076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:27.653889713 +0000 UTC m=+6273.800339578" watchObservedRunningTime="2026-10-31 20:05:27.656163076 +0000 UTC m=+6273.802612940"
Oct 31 20:05:27.771346 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:27.771279    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx" podStartSLOduration=1.7712576690000001 podStartE2EDuration="1.771257669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:27.769710112 +0000 UTC m=+6273.916159977" watchObservedRunningTime="2026-10-31 20:05:27.771257669 +0000 UTC m=+6273.917707536"
Oct 31 20:05:38.550508 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:38.550445    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"
Oct 31 20:05:38.550951 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:38.550643    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"
Oct 31 20:05:38.657984 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:38.657946    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:05:46.046685 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:46.046648    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"]
Oct 31 20:05:46.097746 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:46.097713    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-86kbh"]
Oct 31 20:05:47.995849 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:47.995812    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6929e529-bf71-4487-98d3-5e484cf068b6" path="/var/lib/kubelet/pods/6929e529-bf71-4487-98d3-5e484cf068b6/volumes"
Oct 31 20:05:48.660812 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:48.660784    4484 generic.go:356] "Generic (PLEG): container finished" podID="2fd18916-06fc-454a-afd2-c6a736b72331" containerID="415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a" exitCode=0
Oct 31 20:05:48.661080 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:48.660865    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" event={"ID":"2fd18916-06fc-454a-afd2-c6a736b72331","Type":"ContainerDied","Data":"415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a"}
Oct 31 20:05:48.661339 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:48.661322    4484 scope.go:122] "RemoveContainer" containerID="415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a"
Oct 31 20:05:48.662570 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:48.662544    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 20:05:48.662639 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:48.662615    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 20:05:48.662677 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:48.662650    4484 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"}
Oct 31 20:05:49.690169 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:49.690129    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" event={"ID":"2fd18916-06fc-454a-afd2-c6a736b72331","Type":"ContainerStarted","Data":"79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb"}
Oct 31 20:05:50.676598 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:50.676549    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:50.683163 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:50.683126    4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 20:05:50.683518 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:50.683486    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 20:05:51.503914 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:51.503873    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:51.664982 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:51.664944    4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 20:05:51.665278 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:51.665247    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 20:05:52.675507 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:52.673929    4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 20:05:52.675507 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:52.673996    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 20:05:53.681501 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:53.671908    4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 20:05:53.681501 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:53.671975    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 20:05:54.662067 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:54.662014    4484 patch_prober.go:29] interesting pod/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Oct 31 20:05:54.662308 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:54.662089    4484 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Oct 31 20:05:55.092584 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:55.092554    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 20:05:55.102279 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:55.102255    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 20:05:55.469717 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:05:55.469623    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:05:55.665649 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:55.665616    4484 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:55.666653 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:55.666629    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:05:56.587063 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:56.587021    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:05:58.047325 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:58.047282    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"]
Oct 31 20:05:58.098094 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:58.098057    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dntfj"]
Oct 31 20:05:59.993165 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:05:59.993132    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc510fa5-8ea4-45d2-ad15-a8f78f8a3884" path="/var/lib/kubelet/pods/cc510fa5-8ea4-45d2-ad15-a8f78f8a3884/volumes"
Oct 31 20:06:21.209526 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:21.209451    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:06:25.467719 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:06:25.467682    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:06:34.048178 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:34.048147    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"]
Oct 31 20:06:34.099138 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:34.099106    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tztzw"]
Oct 31 20:06:35.530528 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.530493    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"]
Oct 31 20:06:35.534302 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.534286    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:35.639520 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.639453    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zff7k\" (UniqueName: \"kubernetes.io/projected/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-kube-api-access-zff7k\") pod \"maas-api-revoke-sub-fd37c688-jjvlv\" (UID: \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:35.639717 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.639691    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd37c688-jjvlv\" (UID: \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:35.639770 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.639741    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd37c688-jjvlv\" (UID: \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:35.644788 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.644762    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:06:35.644938 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.644869    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:06:35.647249 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.647225    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"]
Oct 31 20:06:35.740915 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.740836    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd37c688-jjvlv\" (UID: \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:35.740915 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.740877    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd37c688-jjvlv\" (UID: \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:35.740915 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.740905    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zff7k\" (UniqueName: \"kubernetes.io/projected/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-kube-api-access-zff7k\") pod \"maas-api-revoke-sub-fd37c688-jjvlv\" (UID: \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:35.741554 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.741532    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd37c688-jjvlv\" (UID: \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:35.743384 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.743365    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd37c688-jjvlv\" (UID: \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:35.799727 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.799650    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zff7k\" (UniqueName: \"kubernetes.io/projected/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-kube-api-access-zff7k\") pod \"maas-api-revoke-sub-fd37c688-jjvlv\" (UID: \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:35.847332 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.847302    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:35.992809 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:35.992771    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d08ea57a-6133-4de8-9827-53adfadac0c1" path="/var/lib/kubelet/pods/d08ea57a-6133-4de8-9827-53adfadac0c1/volumes"
Oct 31 20:06:36.220567 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:36.220537    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"]
Oct 31 20:06:36.221275 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:06:36.221251    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf591d17f_8fa0_4587_afdd_1afcdaa5c2c1.slice/crio-f9290765ab69dea3d6bf0582e7743ee7184fa3816d70eedeefa5f6f056b21344 WatchSource:0}: Error finding container f9290765ab69dea3d6bf0582e7743ee7184fa3816d70eedeefa5f6f056b21344: Status 404 returned error can't find the container with id f9290765ab69dea3d6bf0582e7743ee7184fa3816d70eedeefa5f6f056b21344
Oct 31 20:06:36.955272 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:36.955243    4484 generic.go:356] "Generic (PLEG): container finished" podID="f591d17f-8fa0-4587-afdd-1afcdaa5c2c1" containerID="9c844cd07f47b3bd7d0bd6875028073397624b7143d4d4896cd790e8d65a9280" exitCode=0
Oct 31 20:06:36.955685 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:36.955325    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv" event={"ID":"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1","Type":"ContainerDied","Data":"9c844cd07f47b3bd7d0bd6875028073397624b7143d4d4896cd790e8d65a9280"}
Oct 31 20:06:36.955685 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:36.955386    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv" event={"ID":"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1","Type":"ContainerStarted","Data":"f9290765ab69dea3d6bf0582e7743ee7184fa3816d70eedeefa5f6f056b21344"}
Oct 31 20:06:38.104313 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.104290    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:38.160818 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.160786    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-kube-api-access-zff7k\" (UniqueName: \"kubernetes.io/projected/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-kube-api-access-zff7k\") pod \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\" (UID: \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\") "
Oct 31 20:06:38.161033 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.160873    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-xks-ca\") pod \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\" (UID: \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\") "
Oct 31 20:06:38.161033 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.160897    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-openshift-ca\") pod \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\" (UID: \"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1\") "
Oct 31 20:06:38.161313 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.161292    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-openshift-ca" pod "f591d17f-8fa0-4587-afdd-1afcdaa5c2c1" (UID: "f591d17f-8fa0-4587-afdd-1afcdaa5c2c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:38.163110 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.163086    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-xks-ca" pod "f591d17f-8fa0-4587-afdd-1afcdaa5c2c1" (UID: "f591d17f-8fa0-4587-afdd-1afcdaa5c2c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:38.163209 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.163127    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-kube-api-access-zff7k" pod "f591d17f-8fa0-4587-afdd-1afcdaa5c2c1" (UID: "f591d17f-8fa0-4587-afdd-1afcdaa5c2c1"). InnerVolumeSpecName "kube-api-access-zff7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:38.262146 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.262117    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zff7k\" (UniqueName: \"kubernetes.io/projected/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-kube-api-access-zff7k\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:06:38.262146 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.262147    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:06:38.262355 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.262157    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:06:38.967090 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.967064    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"
Oct 31 20:06:38.967275 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.967085    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv" event={"ID":"f591d17f-8fa0-4587-afdd-1afcdaa5c2c1","Type":"ContainerDied","Data":"f9290765ab69dea3d6bf0582e7743ee7184fa3816d70eedeefa5f6f056b21344"}
Oct 31 20:06:38.967275 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:38.967155    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9290765ab69dea3d6bf0582e7743ee7184fa3816d70eedeefa5f6f056b21344"
Oct 31 20:06:44.047501 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:44.047452    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"]
Oct 31 20:06:44.098235 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:44.098196    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-bl9t6"]
Oct 31 20:06:45.992184 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:45.992146    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf2bc07-7df8-4120-a99f-6639134a5279" path="/var/lib/kubelet/pods/9bf2bc07-7df8-4120-a99f-6639134a5279/volumes"
Oct 31 20:06:50.368936 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:50.368885    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:06:53.446192 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:53.446153    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"]
Oct 31 20:06:53.546607 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:53.546568    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"]
Oct 31 20:06:55.469223 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:06:55.469188    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:06:56.480019 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:56.479989    4484 scope.go:122] "RemoveContainer" containerID="e4bbf35b6ece31f239f2f3ee2f88ec63ae834147a7cb861910e9cbf8420f1b5f"
Oct 31 20:06:56.489276 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:56.489256    4484 scope.go:122] "RemoveContainer" containerID="866cecb28958805984a6f1c921422d5a0a3b1265f76f407f5c48f7ffdb544f01"
Oct 31 20:06:58.446776 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:58.446723    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx" podUID="b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee" containerName="payload-processing" containerID="cri-o://804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9" gracePeriod=25
Oct 31 20:06:58.547672 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:58.547625    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc" podUID="9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc" containerName="payload-pre-processing" containerID="cri-o://9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc" gracePeriod=25
Oct 31 20:06:58.548907 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:06:58.548869    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx" podUID="b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:05.659036 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:05.658991    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:07:08.548991 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:08.548949    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx" podUID="b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:08.549413 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:08.548957    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc" podUID="9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:18.548312 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:07:18.548269    4484 logging.go:55] [core] [Channel #24205 SubChannel #24208]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.28:50052", ServerName: "10.133.1.28:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.28:50052: connect: connection refused"
Oct 31 20:07:18.549033 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.549004    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc" podUID="9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:07:18.705599 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.705538    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"
Oct 31 20:07:18.795372 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.795346    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"
Oct 31 20:07:18.862492 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.862444    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-kube-api-access-29fqm\" (UniqueName: \"kubernetes.io/projected/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-kube-api-access-29fqm\") pod \"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee\" (UID: \"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee\") "
Oct 31 20:07:18.862678 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.862617    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-plugins-config-volume\") pod \"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee\" (UID: \"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee\") "
Oct 31 20:07:18.862967 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.862941    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-plugins-config-volume" pod "b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee" (UID: "b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:18.864657 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.864638    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-kube-api-access-29fqm" pod "b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee" (UID: "b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee"). InnerVolumeSpecName "kube-api-access-29fqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:18.964206 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.964123    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-kube-api-access-dlxds\" (UniqueName: \"kubernetes.io/projected/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-kube-api-access-dlxds\") pod \"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc\" (UID: \"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc\") "
Oct 31 20:07:18.964404 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.964216    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-plugins-config-volume\") pod \"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc\" (UID: \"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc\") "
Oct 31 20:07:18.964497 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.964443    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:18.964497 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.964459    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29fqm\" (UniqueName: \"kubernetes.io/projected/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee-kube-api-access-29fqm\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:18.964705 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.964674    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-plugins-config-volume" pod "9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc" (UID: "9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:18.966458 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:18.966434    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-kube-api-access-dlxds" pod "9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc" (UID: "9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc"). InnerVolumeSpecName "kube-api-access-dlxds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:19.055230 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.055185    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"]
Oct 31 20:07:19.057429 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.057400    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-hmc7x"]
Oct 31 20:07:19.065240 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.065205    4484 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-plugins-config-volume\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:19.065240 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.065237    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxds\" (UniqueName: \"kubernetes.io/projected/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc-kube-api-access-dlxds\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:19.179908 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.179876    4484 generic.go:356] "Generic (PLEG): container finished" podID="9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc" containerID="9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc" exitCode=0
Oct 31 20:07:19.180134 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.179938    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"
Oct 31 20:07:19.180134 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.179967    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc" event={"ID":"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc","Type":"ContainerDied","Data":"9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc"}
Oct 31 20:07:19.180134 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.180030    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc" event={"ID":"9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc","Type":"ContainerDied","Data":"97908e18f23c190c04b27a943408b1b015d564a7e5b42dd91068c4368e87c37e"}
Oct 31 20:07:19.180134 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.180062    4484 scope.go:122] "RemoveContainer" containerID="9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc"
Oct 31 20:07:19.181595 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.181570    4484 generic.go:356] "Generic (PLEG): container finished" podID="b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee" containerID="804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9" exitCode=0
Oct 31 20:07:19.181717 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.181639    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"
Oct 31 20:07:19.181803 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.181769    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx" event={"ID":"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee","Type":"ContainerDied","Data":"804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9"}
Oct 31 20:07:19.181852 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.181834    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx" event={"ID":"b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee","Type":"ContainerDied","Data":"ff217d2d31bad07b5fcb877610b8bd7d76c3a353183b71877dd5a17aee85a4f0"}
Oct 31 20:07:19.190280 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.190254    4484 scope.go:122] "RemoveContainer" containerID="9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc"
Oct 31 20:07:19.190871 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:07:19.190719    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc\": container with ID starting with 9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc not found: ID does not exist" containerID="9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc"
Oct 31 20:07:19.190871 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.190747    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc"} err="failed to get container status \"9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc\": rpc error: code = NotFound desc = could not find container \"9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc\": container with ID starting with 9ddf8ca845e0259c2cc8a74a0ebff6ea4b289298ee555aadf83a0a8dc8e528cc not found: ID does not exist"
Oct 31 20:07:19.190871 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.190776    4484 scope.go:122] "RemoveContainer" containerID="804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9"
Oct 31 20:07:19.199800 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.199779    4484 scope.go:122] "RemoveContainer" containerID="804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9"
Oct 31 20:07:19.200142 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:07:19.200118    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9\": container with ID starting with 804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9 not found: ID does not exist" containerID="804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9"
Oct 31 20:07:19.200212 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.200151    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9"} err="failed to get container status \"804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9\": rpc error: code = NotFound desc = could not find container \"804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9\": container with ID starting with 804532058ce42105d7f5b545f25857af62cea67bd4d7206bd9864f88cca873c9 not found: ID does not exist"
Oct 31 20:07:19.349948 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.349910    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"]
Oct 31 20:07:19.354582 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.354543    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-42030c-865db9jdpc"]
Oct 31 20:07:19.466458 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.466428    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"]
Oct 31 20:07:19.472459 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.472434    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx"]
Oct 31 20:07:19.548090 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.548049    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-42030c-5bc5b78cf57xmx" podUID="b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.28:50052\" within 1s: context deadline exceeded"
Oct 31 20:07:19.996518 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.996461    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d883f9f-105c-4e12-a86c-1e31216c932b" path="/var/lib/kubelet/pods/0d883f9f-105c-4e12-a86c-1e31216c932b/volumes"
Oct 31 20:07:19.997064 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.997041    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc" path="/var/lib/kubelet/pods/9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc/volumes"
Oct 31 20:07:19.997456 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:19.997437    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee" path="/var/lib/kubelet/pods/b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee/volumes"
Oct 31 20:07:25.468240 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:07:25.468205    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:07:32.222768 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:32.222733    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"]
Oct 31 20:07:32.223255 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:32.222936    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" containerName="istio-proxy" containerID="cri-o://79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb" gracePeriod=30
Oct 31 20:07:33.798548 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:33.798511    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"]
Oct 31 20:07:33.849373 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:33.849336    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v99nw"]
Oct 31 20:07:33.992885 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:33.992852    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f59c66e-9739-4d5f-a848-65b4b350f640" path="/var/lib/kubelet/pods/0f59c66e-9739-4d5f-a848-65b4b350f640/volumes"
Oct 31 20:07:37.474763 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.474737    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:07:37.552299 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552259    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-certs\") pod \"2fd18916-06fc-454a-afd2-c6a736b72331\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") "
Oct 31 20:07:37.552537 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552313    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-socket\") pod \"2fd18916-06fc-454a-afd2-c6a736b72331\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") "
Oct 31 20:07:37.552537 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552368    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-istio-token\" (UniqueName: \"kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-istio-token\") pod \"2fd18916-06fc-454a-afd2-c6a736b72331\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") "
Oct 31 20:07:37.552537 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552391    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-envoy\") pod \"2fd18916-06fc-454a-afd2-c6a736b72331\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") "
Oct 31 20:07:37.552537 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552415    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fd18916-06fc-454a-afd2-c6a736b72331-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd18916-06fc-454a-afd2-c6a736b72331-istio-podinfo\") pod \"2fd18916-06fc-454a-afd2-c6a736b72331\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") "
Oct 31 20:07:37.552537 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552453    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-credential-socket\") pod \"2fd18916-06fc-454a-afd2-c6a736b72331\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") "
Oct 31 20:07:37.552821 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552552    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-data\") pod \"2fd18916-06fc-454a-afd2-c6a736b72331\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") "
Oct 31 20:07:37.552821 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552629    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd18916-06fc-454a-afd2-c6a736b72331-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd18916-06fc-454a-afd2-c6a736b72331-istiod-ca-cert\") pod \"2fd18916-06fc-454a-afd2-c6a736b72331\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") "
Oct 31 20:07:37.552821 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552638    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-certs" pod "2fd18916-06fc-454a-afd2-c6a736b72331" (UID: "2fd18916-06fc-454a-afd2-c6a736b72331"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:37.552821 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552648    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-socket" pod "2fd18916-06fc-454a-afd2-c6a736b72331" (UID: "2fd18916-06fc-454a-afd2-c6a736b72331"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:37.552821 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552665    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-kube-api-access-2jwzh\" (UniqueName: \"kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-kube-api-access-2jwzh\") pod \"2fd18916-06fc-454a-afd2-c6a736b72331\" (UID: \"2fd18916-06fc-454a-afd2-c6a736b72331\") "
Oct 31 20:07:37.553049 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552825    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-credential-socket" pod "2fd18916-06fc-454a-afd2-c6a736b72331" (UID: "2fd18916-06fc-454a-afd2-c6a736b72331"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:37.553049 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552879    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-data" pod "2fd18916-06fc-454a-afd2-c6a736b72331" (UID: "2fd18916-06fc-454a-afd2-c6a736b72331"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:37.553049 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.552987    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd18916-06fc-454a-afd2-c6a736b72331-istiod-ca-cert" pod "2fd18916-06fc-454a-afd2-c6a736b72331" (UID: "2fd18916-06fc-454a-afd2-c6a736b72331"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:37.553165 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.553066    4484 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.553165 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.553088    4484 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-workload-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.553165 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.553102    4484 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-credential-socket\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.553165 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.553115    4484 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-data\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.554950 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.554923    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-envoy" pod "2fd18916-06fc-454a-afd2-c6a736b72331" (UID: "2fd18916-06fc-454a-afd2-c6a736b72331"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:37.554950 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.554934    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-kube-api-access-2jwzh" pod "2fd18916-06fc-454a-afd2-c6a736b72331" (UID: "2fd18916-06fc-454a-afd2-c6a736b72331"). InnerVolumeSpecName "kube-api-access-2jwzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:37.555228 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.555212    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-istio-token" pod "2fd18916-06fc-454a-afd2-c6a736b72331" (UID: "2fd18916-06fc-454a-afd2-c6a736b72331"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:37.555279 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.555233    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fd18916-06fc-454a-afd2-c6a736b72331-istio-podinfo" pod "2fd18916-06fc-454a-afd2-c6a736b72331" (UID: "2fd18916-06fc-454a-afd2-c6a736b72331"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:07:37.654427 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.654394    4484 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd18916-06fc-454a-afd2-c6a736b72331-istiod-ca-cert\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.654427 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.654424    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jwzh\" (UniqueName: \"kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-kube-api-access-2jwzh\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.654427 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.654435    4484 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd18916-06fc-454a-afd2-c6a736b72331-istio-token\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.654718 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.654446    4484 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd18916-06fc-454a-afd2-c6a736b72331-istio-envoy\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.654718 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:37.654463    4484 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd18916-06fc-454a-afd2-c6a736b72331-istio-podinfo\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:07:38.052801 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.052765    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"]
Oct 31 20:07:38.054843 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.054819    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2z4vd"]
Oct 31 20:07:38.287122 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.287037    4484 generic.go:356] "Generic (PLEG): container finished" podID="2fd18916-06fc-454a-afd2-c6a736b72331" containerID="79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb" exitCode=0
Oct 31 20:07:38.287122 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.287086    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"
Oct 31 20:07:38.287320 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.287117    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" event={"ID":"2fd18916-06fc-454a-afd2-c6a736b72331","Type":"ContainerDied","Data":"79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb"}
Oct 31 20:07:38.287320 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.287176    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv" event={"ID":"2fd18916-06fc-454a-afd2-c6a736b72331","Type":"ContainerDied","Data":"4d7b9b3753008a140a7e5e3423a4a82512d334b79c2179018b6d51ac05348327"}
Oct 31 20:07:38.287320 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.287203    4484 scope.go:122] "RemoveContainer" containerID="79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb"
Oct 31 20:07:38.296495 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.296445    4484 scope.go:122] "RemoveContainer" containerID="415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a"
Oct 31 20:07:38.305652 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.305629    4484 scope.go:122] "RemoveContainer" containerID="79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb"
Oct 31 20:07:38.305954 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:07:38.305936    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb\": container with ID starting with 79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb not found: ID does not exist" containerID="79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb"
Oct 31 20:07:38.305994 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.305967    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb"} err="failed to get container status \"79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb\": rpc error: code = NotFound desc = could not find container \"79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb\": container with ID starting with 79f535e9abc7f9cf1ea5ba18ff383d2be0b67d316d3b7962ab4e9f4b11aa8ddb not found: ID does not exist"
Oct 31 20:07:38.305994 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.305988    4484 scope.go:122] "RemoveContainer" containerID="415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a"
Oct 31 20:07:38.306211 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:07:38.306195    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a\": container with ID starting with 415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a not found: ID does not exist" containerID="415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a"
Oct 31 20:07:38.306266 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.306217    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a"} err="failed to get container status \"415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a\": rpc error: code = NotFound desc = could not find container \"415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a\": container with ID starting with 415707cd936191f55424db1727e70d5ebaecf47584db34e9d9036dcee79f2c6a not found: ID does not exist"
Oct 31 20:07:38.457599 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.457555    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"]
Oct 31 20:07:38.508361 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:38.508331    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-42030c-openshift-default-7c9747985c8xbwv"]
Oct 31 20:07:39.991972 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:39.991942    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" path="/var/lib/kubelet/pods/2fd18916-06fc-454a-afd2-c6a736b72331/volumes"
Oct 31 20:07:39.992405 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:39.992391    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc76d40-1a8f-4314-a90e-953c5ad56414" path="/var/lib/kubelet/pods/3bc76d40-1a8f-4314-a90e-953c5ad56414/volumes"
Oct 31 20:07:48.752682 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:48.752650    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:07:55.468043 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:07:55.467998    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:07:56.517858 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:56.517808    4484 scope.go:122] "RemoveContainer" containerID="c9349c428f397726233aaa5684afa105f0d7a2a254678fe10efa522684141e4d"
Oct 31 20:07:56.526385 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:56.526364    4484 scope.go:122] "RemoveContainer" containerID="323df548e31866dd076ddeb260c1c2ee35be0963862423c897b2d49abc351e06"
Oct 31 20:07:59.055406 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:59.055376    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"]
Oct 31 20:07:59.106239 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:59.106208    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-kgv8g"]
Oct 31 20:07:59.992808 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:07:59.992780    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1b0db3-aa3f-48d8-aa53-82b36064aae2" path="/var/lib/kubelet/pods/ab1b0db3-aa3f-48d8-aa53-82b36064aae2/volumes"
Oct 31 20:08:16.048861 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:08:16.048778    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"]
Oct 31 20:08:16.099312 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:08:16.099276    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vmbx2"]
Oct 31 20:08:17.992157 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:08:17.992126    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec3bb4e-3275-4f47-a371-87e872f6e7db" path="/var/lib/kubelet/pods/0ec3bb4e-3275-4f47-a371-87e872f6e7db/volumes"
Oct 31 20:08:25.469842 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:08:25.469801    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:08:55.469275 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:08:55.469238    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:08:56.560459 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:08:56.560426    4484 scope.go:122] "RemoveContainer" containerID="52b769812de45a127841d4ba8899da1690667fba9cb1478bffc5ccece70f0851"
Oct 31 20:08:56.569545 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:08:56.569524    4484 scope.go:122] "RemoveContainer" containerID="0f91a46a157b17171791f76f89a4e79b523d7cc1accc614be93d7582892bbd25"
Oct 31 20:08:56.577995 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:08:56.577970    4484 scope.go:122] "RemoveContainer" containerID="1d2d10bd59a4ec6f1e77ece32810c0f9e1f1f51098f94d68ca0f593fdc69119f"
Oct 31 20:09:18.942702 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:18.942664    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vnvmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:09:18.943143 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:18.942731    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podUID="a9f80747-2277-4637-a04a-15207891a808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:09:20.782314 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782282    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"]
Oct 31 20:09:20.782751 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782715    4484 cpu_manager.go:414] "containerMap: removing container" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" containerName="istio-proxy"
Oct 31 20:09:20.782751 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782727    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" containerName="istio-proxy"
Oct 31 20:09:20.782751 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782738    4484 cpu_manager.go:414] "containerMap: removing container" podUID="b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee" containerName="payload-processing"
Oct 31 20:09:20.782751 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782743    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee" containerName="payload-processing"
Oct 31 20:09:20.782888 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782757    4484 cpu_manager.go:414] "containerMap: removing container" podUID="f591d17f-8fa0-4587-afdd-1afcdaa5c2c1" containerName="revoke-keys"
Oct 31 20:09:20.782888 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782762    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f591d17f-8fa0-4587-afdd-1afcdaa5c2c1" containerName="revoke-keys"
Oct 31 20:09:20.782888 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782785    4484 cpu_manager.go:414] "containerMap: removing container" podUID="9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc" containerName="payload-pre-processing"
Oct 31 20:09:20.782888 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782790    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc" containerName="payload-pre-processing"
Oct 31 20:09:20.782888 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782846    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" containerName="istio-proxy"
Oct 31 20:09:20.782888 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782855    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca8d0f5-3040-4ab5-aaa3-ab6383332fdc" containerName="payload-pre-processing"
Oct 31 20:09:20.782888 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782863    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" containerName="istio-proxy"
Oct 31 20:09:20.782888 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782869    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c6f20b-c1fd-49db-bfb2-c5d0fda2bcee" containerName="payload-processing"
Oct 31 20:09:20.782888 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.782875    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="f591d17f-8fa0-4587-afdd-1afcdaa5c2c1" containerName="revoke-keys"
Oct 31 20:09:20.786008 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.785991    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:20.832771 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.832733    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bg9c8\" (UID: \"75a114dc-de87-485d-9f80-cd4173c483eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:20.832965 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.832806    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdkz\" (UniqueName: \"kubernetes.io/projected/75a114dc-de87-485d-9f80-cd4173c483eb-kube-api-access-xxdkz\") pod \"maas-api-revoke-sub-d576e68b-bg9c8\" (UID: \"75a114dc-de87-485d-9f80-cd4173c483eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:20.832965 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.832904    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bg9c8\" (UID: \"75a114dc-de87-485d-9f80-cd4173c483eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:20.892074 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.892038    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:09:20.892074 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.892073    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:09:20.894255 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.894228    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"]
Oct 31 20:09:20.934008 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.933969    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bg9c8\" (UID: \"75a114dc-de87-485d-9f80-cd4173c483eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:20.934224 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.934042    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bg9c8\" (UID: \"75a114dc-de87-485d-9f80-cd4173c483eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:20.934224 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.934127    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdkz\" (UniqueName: \"kubernetes.io/projected/75a114dc-de87-485d-9f80-cd4173c483eb-kube-api-access-xxdkz\") pod \"maas-api-revoke-sub-d576e68b-bg9c8\" (UID: \"75a114dc-de87-485d-9f80-cd4173c483eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:20.934792 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.934767    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bg9c8\" (UID: \"75a114dc-de87-485d-9f80-cd4173c483eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:20.936600 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.936571    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bg9c8\" (UID: \"75a114dc-de87-485d-9f80-cd4173c483eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:20.992572 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:20.992537    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdkz\" (UniqueName: \"kubernetes.io/projected/75a114dc-de87-485d-9f80-cd4173c483eb-kube-api-access-xxdkz\") pod \"maas-api-revoke-sub-d576e68b-bg9c8\" (UID: \"75a114dc-de87-485d-9f80-cd4173c483eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:21.098335 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:21.098247    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:21.204197 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:21.204162    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:09:21.476466 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:21.476440    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"]
Oct 31 20:09:21.478386 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:09:21.478359    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a114dc_de87_485d_9f80_cd4173c483eb.slice/crio-b7e17b9c20834801544c951e253860627399964b1ca8155415c3a2fbf69bf2c8 WatchSource:0}: Error finding container b7e17b9c20834801544c951e253860627399964b1ca8155415c3a2fbf69bf2c8: Status 404 returned error can't find the container with id b7e17b9c20834801544c951e253860627399964b1ca8155415c3a2fbf69bf2c8
Oct 31 20:09:21.830653 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:21.830608    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8" event={"ID":"75a114dc-de87-485d-9f80-cd4173c483eb","Type":"ContainerStarted","Data":"02afe16c4c26d0e901de3a786401573c1bce1b6a60cad380cbc63293ae18ced8"}
Oct 31 20:09:21.831072 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:21.830664    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8" event={"ID":"75a114dc-de87-485d-9f80-cd4173c483eb","Type":"ContainerStarted","Data":"b7e17b9c20834801544c951e253860627399964b1ca8155415c3a2fbf69bf2c8"}
Oct 31 20:09:21.947795 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:21.947742    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8" podStartSLOduration=1.9477301649999998 podStartE2EDuration="1.947730165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:09:21.946062881 +0000 UTC m=+6508.092512743" watchObservedRunningTime="2026-10-31 20:09:21.947730165 +0000 UTC m=+6508.094180029"
Oct 31 20:09:22.836802 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:22.836770    4484 generic.go:356] "Generic (PLEG): container finished" podID="75a114dc-de87-485d-9f80-cd4173c483eb" containerID="02afe16c4c26d0e901de3a786401573c1bce1b6a60cad380cbc63293ae18ced8" exitCode=0
Oct 31 20:09:22.837207 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:22.836820    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8" event={"ID":"75a114dc-de87-485d-9f80-cd4173c483eb","Type":"ContainerDied","Data":"02afe16c4c26d0e901de3a786401573c1bce1b6a60cad380cbc63293ae18ced8"}
Oct 31 20:09:23.975642 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:23.975618    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:24.066998 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:24.066961    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-openshift-ca\") pod \"75a114dc-de87-485d-9f80-cd4173c483eb\" (UID: \"75a114dc-de87-485d-9f80-cd4173c483eb\") "
Oct 31 20:09:24.067175 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:24.067016    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a114dc-de87-485d-9f80-cd4173c483eb-kube-api-access-xxdkz\" (UniqueName: \"kubernetes.io/projected/75a114dc-de87-485d-9f80-cd4173c483eb-kube-api-access-xxdkz\") pod \"75a114dc-de87-485d-9f80-cd4173c483eb\" (UID: \"75a114dc-de87-485d-9f80-cd4173c483eb\") "
Oct 31 20:09:24.067175 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:24.067049    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-xks-ca\") pod \"75a114dc-de87-485d-9f80-cd4173c483eb\" (UID: \"75a114dc-de87-485d-9f80-cd4173c483eb\") "
Oct 31 20:09:24.067387 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:24.067342    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-openshift-ca" pod "75a114dc-de87-485d-9f80-cd4173c483eb" (UID: "75a114dc-de87-485d-9f80-cd4173c483eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:09:24.069361 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:24.069328    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-xks-ca" pod "75a114dc-de87-485d-9f80-cd4173c483eb" (UID: "75a114dc-de87-485d-9f80-cd4173c483eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:09:24.069504 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:24.069429    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a114dc-de87-485d-9f80-cd4173c483eb-kube-api-access-xxdkz" pod "75a114dc-de87-485d-9f80-cd4173c483eb" (UID: "75a114dc-de87-485d-9f80-cd4173c483eb"). InnerVolumeSpecName "kube-api-access-xxdkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:09:24.168118 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:24.168019    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:09:24.168118 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:24.168057    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxdkz\" (UniqueName: \"kubernetes.io/projected/75a114dc-de87-485d-9f80-cd4173c483eb-kube-api-access-xxdkz\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:09:24.168118 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:24.168068    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a114dc-de87-485d-9f80-cd4173c483eb-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:09:24.848526 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:24.848464    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8" event={"ID":"75a114dc-de87-485d-9f80-cd4173c483eb","Type":"ContainerDied","Data":"b7e17b9c20834801544c951e253860627399964b1ca8155415c3a2fbf69bf2c8"}
Oct 31 20:09:24.848719 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:24.848582    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7e17b9c20834801544c951e253860627399964b1ca8155415c3a2fbf69bf2c8"
Oct 31 20:09:24.848719 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:24.848506    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"
Oct 31 20:09:25.468065 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:09:25.468028    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:09:33.607662 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:33.607622    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:09:37.046837 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:37.046754    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"]
Oct 31 20:09:37.097601 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:37.097566    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-5mmz2"]
Oct 31 20:09:37.992435 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:37.992392    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2fecf6d-0090-4a15-971b-729227ce1de2" path="/var/lib/kubelet/pods/e2fecf6d-0090-4a15-971b-729227ce1de2/volumes"
Oct 31 20:09:42.031392 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.031365    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:09:42.180377 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.180343    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"]
Oct 31 20:09:42.180838 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.180820    4484 cpu_manager.go:414] "containerMap: removing container" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" containerName="istio-proxy"
Oct 31 20:09:42.180936 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.180840    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd18916-06fc-454a-afd2-c6a736b72331" containerName="istio-proxy"
Oct 31 20:09:42.180936 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.180878    4484 cpu_manager.go:414] "containerMap: removing container" podUID="75a114dc-de87-485d-9f80-cd4173c483eb" containerName="revoke-keys"
Oct 31 20:09:42.180936 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.180886    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a114dc-de87-485d-9f80-cd4173c483eb" containerName="revoke-keys"
Oct 31 20:09:42.181105 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.180993    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a114dc-de87-485d-9f80-cd4173c483eb" containerName="revoke-keys"
Oct 31 20:09:42.185240 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.185218    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:42.240999 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.240969    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"]
Oct 31 20:09:42.289963 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.289888    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:09:42.289963 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.289890    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:09:42.293357 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.293335    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6lmc8\" (UID: \"3b382a5d-b348-4e25-9e17-92c1000de84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:42.293418 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.293389    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpqt7\" (UniqueName: \"kubernetes.io/projected/3b382a5d-b348-4e25-9e17-92c1000de84d-kube-api-access-qpqt7\") pod \"maas-api-revoke-sub-d2bc93bc-6lmc8\" (UID: \"3b382a5d-b348-4e25-9e17-92c1000de84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:42.293574 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.293556    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6lmc8\" (UID: \"3b382a5d-b348-4e25-9e17-92c1000de84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:42.394261 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.394222    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpqt7\" (UniqueName: \"kubernetes.io/projected/3b382a5d-b348-4e25-9e17-92c1000de84d-kube-api-access-qpqt7\") pod \"maas-api-revoke-sub-d2bc93bc-6lmc8\" (UID: \"3b382a5d-b348-4e25-9e17-92c1000de84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:42.394442 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.394280    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6lmc8\" (UID: \"3b382a5d-b348-4e25-9e17-92c1000de84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:42.394442 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.394336    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6lmc8\" (UID: \"3b382a5d-b348-4e25-9e17-92c1000de84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:42.394949 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.394928    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6lmc8\" (UID: \"3b382a5d-b348-4e25-9e17-92c1000de84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:42.396796 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.396772    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-6lmc8\" (UID: \"3b382a5d-b348-4e25-9e17-92c1000de84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:42.452657 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.452621    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpqt7\" (UniqueName: \"kubernetes.io/projected/3b382a5d-b348-4e25-9e17-92c1000de84d-kube-api-access-qpqt7\") pod \"maas-api-revoke-sub-d2bc93bc-6lmc8\" (UID: \"3b382a5d-b348-4e25-9e17-92c1000de84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:42.497561 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.497517    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:42.865763 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.865741    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"]
Oct 31 20:09:42.870923 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:09:42.867630    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b382a5d_b348_4e25_9e17_92c1000de84d.slice/crio-54fdaec0f9282ca62428f8ac1da2c9acf8fdaaf3e42554f40139a9a8b1a6afaf WatchSource:0}: Error finding container 54fdaec0f9282ca62428f8ac1da2c9acf8fdaaf3e42554f40139a9a8b1a6afaf: Status 404 returned error can't find the container with id 54fdaec0f9282ca62428f8ac1da2c9acf8fdaaf3e42554f40139a9a8b1a6afaf
Oct 31 20:09:42.963205 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:42.963169    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8" event={"ID":"3b382a5d-b348-4e25-9e17-92c1000de84d","Type":"ContainerStarted","Data":"54fdaec0f9282ca62428f8ac1da2c9acf8fdaaf3e42554f40139a9a8b1a6afaf"}
Oct 31 20:09:43.970247 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:43.970160    4484 generic.go:356] "Generic (PLEG): container finished" podID="3b382a5d-b348-4e25-9e17-92c1000de84d" containerID="acf5cd64f460ed370daceb24c25d92564f2f07f952c60d8fa53ef02b30dea396" exitCode=0
Oct 31 20:09:43.970247 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:43.970201    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8" event={"ID":"3b382a5d-b348-4e25-9e17-92c1000de84d","Type":"ContainerDied","Data":"acf5cd64f460ed370daceb24c25d92564f2f07f952c60d8fa53ef02b30dea396"}
Oct 31 20:09:45.105897 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.105873    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:45.220736 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.220705    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-xks-ca\") pod \"3b382a5d-b348-4e25-9e17-92c1000de84d\" (UID: \"3b382a5d-b348-4e25-9e17-92c1000de84d\") "
Oct 31 20:09:45.220924 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.220795    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b382a5d-b348-4e25-9e17-92c1000de84d-kube-api-access-qpqt7\" (UniqueName: \"kubernetes.io/projected/3b382a5d-b348-4e25-9e17-92c1000de84d-kube-api-access-qpqt7\") pod \"3b382a5d-b348-4e25-9e17-92c1000de84d\" (UID: \"3b382a5d-b348-4e25-9e17-92c1000de84d\") "
Oct 31 20:09:45.220924 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.220869    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-openshift-ca\") pod \"3b382a5d-b348-4e25-9e17-92c1000de84d\" (UID: \"3b382a5d-b348-4e25-9e17-92c1000de84d\") "
Oct 31 20:09:45.221225 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.221197    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-openshift-ca" pod "3b382a5d-b348-4e25-9e17-92c1000de84d" (UID: "3b382a5d-b348-4e25-9e17-92c1000de84d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:09:45.223339 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.223311    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-xks-ca" pod "3b382a5d-b348-4e25-9e17-92c1000de84d" (UID: "3b382a5d-b348-4e25-9e17-92c1000de84d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:09:45.223661 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.223637    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b382a5d-b348-4e25-9e17-92c1000de84d-kube-api-access-qpqt7" pod "3b382a5d-b348-4e25-9e17-92c1000de84d" (UID: "3b382a5d-b348-4e25-9e17-92c1000de84d"). InnerVolumeSpecName "kube-api-access-qpqt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:09:45.321576 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.321540    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpqt7\" (UniqueName: \"kubernetes.io/projected/3b382a5d-b348-4e25-9e17-92c1000de84d-kube-api-access-qpqt7\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:09:45.321576 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.321572    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-openshift-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:09:45.321576 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.321583    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b382a5d-b348-4e25-9e17-92c1000de84d-maas-api-xks-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:09:45.981025 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.980996    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"
Oct 31 20:09:45.981210 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.981036    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8" event={"ID":"3b382a5d-b348-4e25-9e17-92c1000de84d","Type":"ContainerDied","Data":"54fdaec0f9282ca62428f8ac1da2c9acf8fdaaf3e42554f40139a9a8b1a6afaf"}
Oct 31 20:09:45.981210 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:45.981081    4484 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54fdaec0f9282ca62428f8ac1da2c9acf8fdaaf3e42554f40139a9a8b1a6afaf"
Oct 31 20:09:48.045593 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:48.045556    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"]
Oct 31 20:09:48.096429 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:48.096401    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vz6rq"]
Oct 31 20:09:49.992722 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:49.992680    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a54f96a-7f15-4b54-a81c-f721561c8def" path="/var/lib/kubelet/pods/7a54f96a-7f15-4b54-a81c-f721561c8def/volumes"
Oct 31 20:09:50.376387 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:50.376291    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:09:55.229942 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.229907    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"]
Oct 31 20:09:55.230341 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.230328    4484 cpu_manager.go:414] "containerMap: removing container" podUID="3b382a5d-b348-4e25-9e17-92c1000de84d" containerName="revoke-keys"
Oct 31 20:09:55.230385 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.230341    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b382a5d-b348-4e25-9e17-92c1000de84d" containerName="revoke-keys"
Oct 31 20:09:55.230420 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.230407    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b382a5d-b348-4e25-9e17-92c1000de84d" containerName="revoke-keys"
Oct 31 20:09:55.233715 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.233699    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.318058 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.318023    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa681c0-128b-49f6-8d12-deaa7629fbca-openshift-service-ca\") pod \"maas-api-5cc786fc78-9ppdj\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.318304 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.318229    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmtrn\" (UniqueName: \"kubernetes.io/projected/baa681c0-128b-49f6-8d12-deaa7629fbca-kube-api-access-nmtrn\") pod \"maas-api-5cc786fc78-9ppdj\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.318304 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.318258    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-maas-api-tls\") pod \"maas-api-5cc786fc78-9ppdj\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.318304 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.318299    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-metrics-certs\") pod \"maas-api-5cc786fc78-9ppdj\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.339272 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.339248    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 20:09:55.339434 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.339356    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 20:09:55.339434 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.339364    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-czkfc\""
Oct 31 20:09:55.341563 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.341542    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"]
Oct 31 20:09:55.419163 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.419134    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-metrics-certs\") pod \"maas-api-5cc786fc78-9ppdj\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.419390 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.419182    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa681c0-128b-49f6-8d12-deaa7629fbca-openshift-service-ca\") pod \"maas-api-5cc786fc78-9ppdj\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.419390 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.419290    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtrn\" (UniqueName: \"kubernetes.io/projected/baa681c0-128b-49f6-8d12-deaa7629fbca-kube-api-access-nmtrn\") pod \"maas-api-5cc786fc78-9ppdj\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.419390 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.419309    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-maas-api-tls\") pod \"maas-api-5cc786fc78-9ppdj\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.419934 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.419907    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa681c0-128b-49f6-8d12-deaa7629fbca-openshift-service-ca\") pod \"maas-api-5cc786fc78-9ppdj\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.422053 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.422025    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-metrics-certs\") pod \"maas-api-5cc786fc78-9ppdj\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.422152 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.422132    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-maas-api-tls\") pod \"maas-api-5cc786fc78-9ppdj\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.467175 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:09:55.467142    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:09:55.480246 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.480182    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmtrn\" (UniqueName: \"kubernetes.io/projected/baa681c0-128b-49f6-8d12-deaa7629fbca-kube-api-access-nmtrn\") pod \"maas-api-5cc786fc78-9ppdj\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") " pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.546923 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.546894    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:55.920590 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.920564    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"]
Oct 31 20:09:55.922540 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:09:55.922515    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa681c0_128b_49f6_8d12_deaa7629fbca.slice/crio-e6b0c179c87c0377fa314662d8c08b57d97d04ec99f78cd4e7a0023e145272b2 WatchSource:0}: Error finding container e6b0c179c87c0377fa314662d8c08b57d97d04ec99f78cd4e7a0023e145272b2: Status 404 returned error can't find the container with id e6b0c179c87c0377fa314662d8c08b57d97d04ec99f78cd4e7a0023e145272b2
Oct 31 20:09:55.923874 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:55.923855    4484 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 20:09:56.030009 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:56.029974    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj" event={"ID":"baa681c0-128b-49f6-8d12-deaa7629fbca","Type":"ContainerStarted","Data":"e6b0c179c87c0377fa314662d8c08b57d97d04ec99f78cd4e7a0023e145272b2"}
Oct 31 20:09:56.607922 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:56.607889    4484 scope.go:122] "RemoveContainer" containerID="39532a99a6ef9cf77943d3ed48f47522477fc3d2f88f30b89c8155b5eb0d8927"
Oct 31 20:09:56.619650 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:56.619629    4484 scope.go:122] "RemoveContainer" containerID="d53c1b6ed2a8dc2b573b2b5f4b28b41c821cd946faa68a745b3be26c046fabaf"
Oct 31 20:09:56.627662 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:56.627638    4484 scope.go:122] "RemoveContainer" containerID="9bca35e5e50a8f0a3c0b6359a90e1b41eca747919eebbe8dd763ab18c0ea09fe"
Oct 31 20:09:57.036081 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:57.036046    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj" event={"ID":"baa681c0-128b-49f6-8d12-deaa7629fbca","Type":"ContainerStarted","Data":"05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6"}
Oct 31 20:09:57.036281 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:57.036164    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:09:57.154955 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:09:57.154848    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj" podStartSLOduration=1.220199811 podStartE2EDuration="2.154834952s" totalImagesPullingTime="934.635141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:09:57.153170858 +0000 UTC m=+6543.299620721" watchObservedRunningTime="2026-10-31 20:09:57.154834952 +0000 UTC m=+6543.301284810"
Oct 31 20:10:03.046885 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:03.046855    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:10:03.971716 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:03.971683    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"]
Oct 31 20:10:03.971927 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:03.971885    4484 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj" podUID="baa681c0-128b-49f6-8d12-deaa7629fbca" containerName="maas-api" containerID="cri-o://05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6" gracePeriod=30
Oct 31 20:10:04.023093 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.023061    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"]
Oct 31 20:10:04.027462 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.027437    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.112531 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.112488    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"]
Oct 31 20:10:04.206067 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.206040    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7d65473-dd86-4929-8f15-9239f45f33db-metrics-certs\") pod \"maas-api-68474f84db-lnkqx\" (UID: \"a7d65473-dd86-4929-8f15-9239f45f33db\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.206067 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.206076    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d65473-dd86-4929-8f15-9239f45f33db-openshift-service-ca\") pod \"maas-api-68474f84db-lnkqx\" (UID: \"a7d65473-dd86-4929-8f15-9239f45f33db\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.206317 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.206213    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz4kt\" (UniqueName: \"kubernetes.io/projected/a7d65473-dd86-4929-8f15-9239f45f33db-kube-api-access-dz4kt\") pod \"maas-api-68474f84db-lnkqx\" (UID: \"a7d65473-dd86-4929-8f15-9239f45f33db\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.206317 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.206278    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7d65473-dd86-4929-8f15-9239f45f33db-maas-api-tls\") pod \"maas-api-68474f84db-lnkqx\" (UID: \"a7d65473-dd86-4929-8f15-9239f45f33db\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.240721 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.240692    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:10:04.307161 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.307131    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-maas-api-tls\") pod \"baa681c0-128b-49f6-8d12-deaa7629fbca\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") "
Oct 31 20:10:04.307336 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.307301    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7d65473-dd86-4929-8f15-9239f45f33db-maas-api-tls\") pod \"maas-api-68474f84db-lnkqx\" (UID: \"a7d65473-dd86-4929-8f15-9239f45f33db\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.307397 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.307375    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7d65473-dd86-4929-8f15-9239f45f33db-metrics-certs\") pod \"maas-api-68474f84db-lnkqx\" (UID: \"a7d65473-dd86-4929-8f15-9239f45f33db\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.307440 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.307404    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d65473-dd86-4929-8f15-9239f45f33db-openshift-service-ca\") pod \"maas-api-68474f84db-lnkqx\" (UID: \"a7d65473-dd86-4929-8f15-9239f45f33db\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.307511 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.307500    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz4kt\" (UniqueName: \"kubernetes.io/projected/a7d65473-dd86-4929-8f15-9239f45f33db-kube-api-access-dz4kt\") pod \"maas-api-68474f84db-lnkqx\" (UID: \"a7d65473-dd86-4929-8f15-9239f45f33db\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.308235 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.308146    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7d65473-dd86-4929-8f15-9239f45f33db-openshift-service-ca\") pod \"maas-api-68474f84db-lnkqx\" (UID: \"a7d65473-dd86-4929-8f15-9239f45f33db\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.309434 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.309413    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-maas-api-tls" pod "baa681c0-128b-49f6-8d12-deaa7629fbca" (UID: "baa681c0-128b-49f6-8d12-deaa7629fbca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:10:04.309909 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.309885    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7d65473-dd86-4929-8f15-9239f45f33db-maas-api-tls\") pod \"maas-api-68474f84db-lnkqx\" (UID: \"a7d65473-dd86-4929-8f15-9239f45f33db\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.310013 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.309956    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7d65473-dd86-4929-8f15-9239f45f33db-metrics-certs\") pod \"maas-api-68474f84db-lnkqx\" (UID: \"a7d65473-dd86-4929-8f15-9239f45f33db\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.366842 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.366809    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz4kt\" (UniqueName: \"kubernetes.io/projected/a7d65473-dd86-4929-8f15-9239f45f33db-kube-api-access-dz4kt\") pod \"maas-api-68474f84db-lnkqx\" (UID: \"a7d65473-dd86-4929-8f15-9239f45f33db\") " pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:04.408655 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.408628    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa681c0-128b-49f6-8d12-deaa7629fbca-kube-api-access-nmtrn\" (UniqueName: \"kubernetes.io/projected/baa681c0-128b-49f6-8d12-deaa7629fbca-kube-api-access-nmtrn\") pod \"baa681c0-128b-49f6-8d12-deaa7629fbca\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") "
Oct 31 20:10:04.408848 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.408672    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-metrics-certs\") pod \"baa681c0-128b-49f6-8d12-deaa7629fbca\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") "
Oct 31 20:10:04.408848 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.408694    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa681c0-128b-49f6-8d12-deaa7629fbca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa681c0-128b-49f6-8d12-deaa7629fbca-openshift-service-ca\") pod \"baa681c0-128b-49f6-8d12-deaa7629fbca\" (UID: \"baa681c0-128b-49f6-8d12-deaa7629fbca\") "
Oct 31 20:10:04.408977 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.408947    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-maas-api-tls\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:10:04.409096 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.409074    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa681c0-128b-49f6-8d12-deaa7629fbca-openshift-service-ca" pod "baa681c0-128b-49f6-8d12-deaa7629fbca" (UID: "baa681c0-128b-49f6-8d12-deaa7629fbca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:10:04.411106 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.411078    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-metrics-certs" pod "baa681c0-128b-49f6-8d12-deaa7629fbca" (UID: "baa681c0-128b-49f6-8d12-deaa7629fbca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:10:04.411236 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.411153    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa681c0-128b-49f6-8d12-deaa7629fbca-kube-api-access-nmtrn" pod "baa681c0-128b-49f6-8d12-deaa7629fbca" (UID: "baa681c0-128b-49f6-8d12-deaa7629fbca"). InnerVolumeSpecName "kube-api-access-nmtrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:10:04.510050 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.510021    4484 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa681c0-128b-49f6-8d12-deaa7629fbca-metrics-certs\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:10:04.510050 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.510048    4484 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa681c0-128b-49f6-8d12-deaa7629fbca-openshift-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:10:04.510264 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.510058    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmtrn\" (UniqueName: \"kubernetes.io/projected/baa681c0-128b-49f6-8d12-deaa7629fbca-kube-api-access-nmtrn\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:10:04.648344 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:04.648300    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:05.021022 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:05.020998    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"]
Oct 31 20:10:05.022614 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:10:05.022578    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d65473_dd86_4929_8f15_9239f45f33db.slice/crio-0ac0b7ac4c4da218df374951d728f0739f52198d49c082e8fdf2275d01280177 WatchSource:0}: Error finding container 0ac0b7ac4c4da218df374951d728f0739f52198d49c082e8fdf2275d01280177: Status 404 returned error can't find the container with id 0ac0b7ac4c4da218df374951d728f0739f52198d49c082e8fdf2275d01280177
Oct 31 20:10:05.076007 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:05.075980    4484 generic.go:356] "Generic (PLEG): container finished" podID="baa681c0-128b-49f6-8d12-deaa7629fbca" containerID="05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6" exitCode=0
Oct 31 20:10:05.076150 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:05.076035    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"
Oct 31 20:10:05.076150 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:05.076065    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj" event={"ID":"baa681c0-128b-49f6-8d12-deaa7629fbca","Type":"ContainerDied","Data":"05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6"}
Oct 31 20:10:05.076150 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:05.076131    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj" event={"ID":"baa681c0-128b-49f6-8d12-deaa7629fbca","Type":"ContainerDied","Data":"e6b0c179c87c0377fa314662d8c08b57d97d04ec99f78cd4e7a0023e145272b2"}
Oct 31 20:10:05.076303 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:05.076160    4484 scope.go:122] "RemoveContainer" containerID="05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6"
Oct 31 20:10:05.077416 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:05.077350    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx" event={"ID":"a7d65473-dd86-4929-8f15-9239f45f33db","Type":"ContainerStarted","Data":"0ac0b7ac4c4da218df374951d728f0739f52198d49c082e8fdf2275d01280177"}
Oct 31 20:10:05.085379 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:05.085363    4484 scope.go:122] "RemoveContainer" containerID="05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6"
Oct 31 20:10:05.085676 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:10:05.085656    4484 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6\": container with ID starting with 05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6 not found: ID does not exist" containerID="05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6"
Oct 31 20:10:05.085770 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:05.085681    4484 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6"} err="failed to get container status \"05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6\": rpc error: code = NotFound desc = could not find container \"05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6\": container with ID starting with 05382e0dd042b465d57e63e49f45429d5f12a03461044e780417b8094d4771b6 not found: ID does not exist"
Oct 31 20:10:05.243113 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:05.243035    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"]
Oct 31 20:10:05.248511 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:05.248469    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cc786fc78-9ppdj"]
Oct 31 20:10:05.992780 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:05.992751    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa681c0-128b-49f6-8d12-deaa7629fbca" path="/var/lib/kubelet/pods/baa681c0-128b-49f6-8d12-deaa7629fbca/volumes"
Oct 31 20:10:06.083534 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:06.083501    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx" event={"ID":"a7d65473-dd86-4929-8f15-9239f45f33db","Type":"ContainerStarted","Data":"be092312c63ed53daf00831b88f2ddf27aa5b20ec2730f5f7779fed72d02409d"}
Oct 31 20:10:06.083721 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:06.083589    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:06.203964 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:06.203869    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx" podStartSLOduration=2.385893328 podStartE2EDuration="3.20385614s" totalImagesPullingTime="817.962812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:10:06.202410702 +0000 UTC m=+6552.348860567" watchObservedRunningTime="2026-10-31 20:10:06.20385614 +0000 UTC m=+6552.350306003"
Oct 31 20:10:07.795960 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:07.795923    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:10:12.095391 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:12.095355    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68474f84db-lnkqx"
Oct 31 20:10:25.467729 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:10:25.467684    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:10:43.787797 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:43.787758    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:10:55.125063 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:55.125034    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 20:10:55.135982 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:55.135959    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 20:10:55.467956 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:10:55.467875    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:10:56.379882 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:56.379845    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:10:56.654564 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:56.654453    4484 scope.go:122] "RemoveContainer" containerID="8f3ca92a77ddb8f753a629d9b199495c498d6b3bbdf54edce6f7f7552a7c7b0f"
Oct 31 20:10:56.663338 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:10:56.663320    4484 scope.go:122] "RemoveContainer" containerID="01770e0e5543f47ec4959ec72a63552d914d256999589bb1faf0e883b6eaaa28"
Oct 31 20:11:20.316962 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:11:20.316924    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:11:25.467659 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:11:25.467621    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:11:33.094206 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:11:33.094171    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:11:39.047431 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:11:39.047395    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"]
Oct 31 20:11:39.098395 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:11:39.098365    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd37c688-jjvlv"]
Oct 31 20:11:39.991935 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:11:39.991900    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f591d17f-8fa0-4587-afdd-1afcdaa5c2c1" path="/var/lib/kubelet/pods/f591d17f-8fa0-4587-afdd-1afcdaa5c2c1/volumes"
Oct 31 20:11:55.467438 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:11:55.467396    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:11:56.688653 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:11:56.688621    4484 scope.go:122] "RemoveContainer" containerID="76a91c28cc7fef243d3dab42787b736ab8b1a3eb395d3abbfe191776e2f69d58"
Oct 31 20:12:05.266902 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:12:05.266862    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:12:15.980937 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:12:15.980900    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:12:17.480499 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:12:17.469445    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-g4xdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:12:17.480499 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:12:17.469594    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" podUID="a7dcf339-7a9f-47df-acef-a2ab94742e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:12:25.478502 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:12:25.477813    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:12:28.717105 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:12:28.717067    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:12:40.710157 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:12:40.710072    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:12:55.467176 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:12:55.467137    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:12:56.709129 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:12:56.709091    4484 scope.go:122] "RemoveContainer" containerID="9c844cd07f47b3bd7d0bd6875028073397624b7143d4d4896cd790e8d65a9280"
Oct 31 20:13:04.284562 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:13:04.284523    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:13:14.956655 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:13:14.956619    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:13:24.894546 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:13:24.894511    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:13:25.468580 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:13:25.468541    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:13:44.606939 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:13:44.606900    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:13:47.502500 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:13:47.501627    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-g4xdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:13:47.502500 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:13:47.502054    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" podUID="a7dcf339-7a9f-47df-acef-a2ab94742e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:13:55.467925 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:13:55.467888    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:14:08.289669 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:14:08.289632    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:14:24.047853 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:14:24.047818    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"]
Oct 31 20:14:24.098534 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:14:24.098502    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bg9c8"]
Oct 31 20:14:25.468788 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:14:25.468751    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:14:25.992883 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:14:25.992836    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a114dc-de87-485d-9f80-cd4173c483eb" path="/var/lib/kubelet/pods/75a114dc-de87-485d-9f80-cd4173c483eb/volumes"
Oct 31 20:14:46.051489 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:14:46.051437    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"]
Oct 31 20:14:46.102348 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:14:46.102303    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-6lmc8"]
Oct 31 20:14:47.992239 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:14:47.992204    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b382a5d-b348-4e25-9e17-92c1000de84d" path="/var/lib/kubelet/pods/3b382a5d-b348-4e25-9e17-92c1000de84d/volumes"
Oct 31 20:14:55.469530 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:14:55.469459    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:15:00.165725 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.165690    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"]
Oct 31 20:15:00.166152 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.166139    4484 cpu_manager.go:414] "containerMap: removing container" podUID="baa681c0-128b-49f6-8d12-deaa7629fbca" containerName="maas-api"
Oct 31 20:15:00.166198 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.166154    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa681c0-128b-49f6-8d12-deaa7629fbca" containerName="maas-api"
Oct 31 20:15:00.166235 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.166225    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa681c0-128b-49f6-8d12-deaa7629fbca" containerName="maas-api"
Oct 31 20:15:00.169310 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.169286    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"
Oct 31 20:15:00.225101 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.225050    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"]
Oct 31 20:15:00.273692 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.273663    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-flwsl\""
Oct 31 20:15:00.308862 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.308826    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z7d7\" (UniqueName: \"kubernetes.io/projected/b42c75fd-c612-494f-9238-41fce8bbf059-kube-api-access-9z7d7\") pod \"maas-api-key-cleanup-29891295-fs7vr\" (UID: \"b42c75fd-c612-494f-9238-41fce8bbf059\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"
Oct 31 20:15:00.308862 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.308864    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42c75fd-c612-494f-9238-41fce8bbf059-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891295-fs7vr\" (UID: \"b42c75fd-c612-494f-9238-41fce8bbf059\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"
Oct 31 20:15:00.410498 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.410436    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z7d7\" (UniqueName: \"kubernetes.io/projected/b42c75fd-c612-494f-9238-41fce8bbf059-kube-api-access-9z7d7\") pod \"maas-api-key-cleanup-29891295-fs7vr\" (UID: \"b42c75fd-c612-494f-9238-41fce8bbf059\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"
Oct 31 20:15:00.410685 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.410511    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42c75fd-c612-494f-9238-41fce8bbf059-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891295-fs7vr\" (UID: \"b42c75fd-c612-494f-9238-41fce8bbf059\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"
Oct 31 20:15:00.411095 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.411075    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42c75fd-c612-494f-9238-41fce8bbf059-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891295-fs7vr\" (UID: \"b42c75fd-c612-494f-9238-41fce8bbf059\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"
Oct 31 20:15:00.470247 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.470167    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z7d7\" (UniqueName: \"kubernetes.io/projected/b42c75fd-c612-494f-9238-41fce8bbf059-kube-api-access-9z7d7\") pod \"maas-api-key-cleanup-29891295-fs7vr\" (UID: \"b42c75fd-c612-494f-9238-41fce8bbf059\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"
Oct 31 20:15:00.490032 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.490003    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"
Oct 31 20:15:00.866773 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:00.866745    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"]
Oct 31 20:15:00.868437 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:15:00.868407    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42c75fd_c612_494f_9238_41fce8bbf059.slice/crio-6e0d95424d0db4b4d044d8362c489c41bf0a88d61486165350cc04953ebe344f WatchSource:0}: Error finding container 6e0d95424d0db4b4d044d8362c489c41bf0a88d61486165350cc04953ebe344f: Status 404 returned error can't find the container with id 6e0d95424d0db4b4d044d8362c489c41bf0a88d61486165350cc04953ebe344f
Oct 31 20:15:01.809011 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:01.808971    4484 generic.go:356] "Generic (PLEG): container finished" podID="b42c75fd-c612-494f-9238-41fce8bbf059" containerID="b58ff31fb515f2d608472f44192a3b667f9b0801b0ae089b7bbd88ab51320d25" exitCode=60
Oct 31 20:15:01.809567 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:01.809034    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr" event={"ID":"b42c75fd-c612-494f-9238-41fce8bbf059","Type":"ContainerDied","Data":"b58ff31fb515f2d608472f44192a3b667f9b0801b0ae089b7bbd88ab51320d25"}
Oct 31 20:15:01.809567 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:01.809088    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr" event={"ID":"b42c75fd-c612-494f-9238-41fce8bbf059","Type":"ContainerStarted","Data":"6e0d95424d0db4b4d044d8362c489c41bf0a88d61486165350cc04953ebe344f"}
Oct 31 20:15:01.809567 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:01.809384    4484 scope.go:122] "RemoveContainer" containerID="b58ff31fb515f2d608472f44192a3b667f9b0801b0ae089b7bbd88ab51320d25"
Oct 31 20:15:02.833502 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:02.826706    4484 generic.go:356] "Generic (PLEG): container finished" podID="b42c75fd-c612-494f-9238-41fce8bbf059" containerID="4ced5aab83abf965ea540e44e862b3a7a44718e6e1e137247004e1956e7f0858" exitCode=60
Oct 31 20:15:02.833502 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:02.826772    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr" event={"ID":"b42c75fd-c612-494f-9238-41fce8bbf059","Type":"ContainerDied","Data":"4ced5aab83abf965ea540e44e862b3a7a44718e6e1e137247004e1956e7f0858"}
Oct 31 20:15:02.833502 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:02.826831    4484 scope.go:122] "RemoveContainer" containerID="b58ff31fb515f2d608472f44192a3b667f9b0801b0ae089b7bbd88ab51320d25"
Oct 31 20:15:02.833502 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:02.827356    4484 scope.go:122] "RemoveContainer" containerID="4ced5aab83abf965ea540e44e862b3a7a44718e6e1e137247004e1956e7f0858"
Oct 31 20:15:02.833502 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:15:02.827675    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891295-fs7vr_odh-ai-gateway-infra(b42c75fd-c612-494f-9238-41fce8bbf059)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr" podUID="b42c75fd-c612-494f-9238-41fce8bbf059"
Oct 31 20:15:03.835492 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:03.834954    4484 scope.go:122] "RemoveContainer" containerID="4ced5aab83abf965ea540e44e862b3a7a44718e6e1e137247004e1956e7f0858"
Oct 31 20:15:03.835492 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:15:03.835202    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891295-fs7vr_odh-ai-gateway-infra(b42c75fd-c612-494f-9238-41fce8bbf059)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr" podUID="b42c75fd-c612-494f-9238-41fce8bbf059"
Oct 31 20:15:03.946675 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:03.946613    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vnvmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:15:03.946851 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:03.946715    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vnvmf" podUID="a9f80747-2277-4637-a04a-15207891a808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:15:13.998272 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:13.998234    4484 scope.go:122] "RemoveContainer" containerID="4ced5aab83abf965ea540e44e862b3a7a44718e6e1e137247004e1956e7f0858"
Oct 31 20:15:14.927048 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:14.926941    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr" event={"ID":"b42c75fd-c612-494f-9238-41fce8bbf059","Type":"ContainerStarted","Data":"977022e2656c6532fa3c836a1692c7ee2ed222e95d5292e9d3714873642d6b67"}
Oct 31 20:15:15.045283 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:15.045225    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr" podStartSLOduration=15.045211211 podStartE2EDuration="15.045211211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:15:15.044460892 +0000 UTC m=+6861.190910757" watchObservedRunningTime="2026-10-31 20:15:15.045211211 +0000 UTC m=+6861.191661076"
Oct 31 20:15:15.934870 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:15.934835    4484 generic.go:356] "Generic (PLEG): container finished" podID="b42c75fd-c612-494f-9238-41fce8bbf059" containerID="977022e2656c6532fa3c836a1692c7ee2ed222e95d5292e9d3714873642d6b67" exitCode=60
Oct 31 20:15:15.935052 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:15.934908    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr" event={"ID":"b42c75fd-c612-494f-9238-41fce8bbf059","Type":"ContainerDied","Data":"977022e2656c6532fa3c836a1692c7ee2ed222e95d5292e9d3714873642d6b67"}
Oct 31 20:15:15.935052 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:15.934976    4484 scope.go:122] "RemoveContainer" containerID="4ced5aab83abf965ea540e44e862b3a7a44718e6e1e137247004e1956e7f0858"
Oct 31 20:15:15.935360 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:15.935339    4484 scope.go:122] "RemoveContainer" containerID="977022e2656c6532fa3c836a1692c7ee2ed222e95d5292e9d3714873642d6b67"
Oct 31 20:15:15.935661 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:15:15.935635    4484 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891295-fs7vr_odh-ai-gateway-infra(b42c75fd-c612-494f-9238-41fce8bbf059)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr" podUID="b42c75fd-c612-494f-9238-41fce8bbf059"
Oct 31 20:15:16.063568 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:16.063535    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"]
Oct 31 20:15:17.072024 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:17.071997    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"
Oct 31 20:15:17.080460 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:17.080419    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42c75fd-c612-494f-9238-41fce8bbf059-kube-api-access-9z7d7\" (UniqueName: \"kubernetes.io/projected/b42c75fd-c612-494f-9238-41fce8bbf059-kube-api-access-9z7d7\") pod \"b42c75fd-c612-494f-9238-41fce8bbf059\" (UID: \"b42c75fd-c612-494f-9238-41fce8bbf059\") "
Oct 31 20:15:17.080673 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:17.080532    4484 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42c75fd-c612-494f-9238-41fce8bbf059-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42c75fd-c612-494f-9238-41fce8bbf059-maas-api-service-ca\") pod \"b42c75fd-c612-494f-9238-41fce8bbf059\" (UID: \"b42c75fd-c612-494f-9238-41fce8bbf059\") "
Oct 31 20:15:17.080871 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:17.080846    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42c75fd-c612-494f-9238-41fce8bbf059-maas-api-service-ca" pod "b42c75fd-c612-494f-9238-41fce8bbf059" (UID: "b42c75fd-c612-494f-9238-41fce8bbf059"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:15:17.082768 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:17.082745    4484 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42c75fd-c612-494f-9238-41fce8bbf059-kube-api-access-9z7d7" pod "b42c75fd-c612-494f-9238-41fce8bbf059" (UID: "b42c75fd-c612-494f-9238-41fce8bbf059"). InnerVolumeSpecName "kube-api-access-9z7d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:15:17.181637 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:17.181605    4484 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42c75fd-c612-494f-9238-41fce8bbf059-maas-api-service-ca\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:15:17.181637 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:17.181634    4484 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z7d7\" (UniqueName: \"kubernetes.io/projected/b42c75fd-c612-494f-9238-41fce8bbf059-kube-api-access-9z7d7\") on node \"ip-10-0-142-120.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 20:15:17.947182 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:17.947149    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr" event={"ID":"b42c75fd-c612-494f-9238-41fce8bbf059","Type":"ContainerDied","Data":"6e0d95424d0db4b4d044d8362c489c41bf0a88d61486165350cc04953ebe344f"}
Oct 31 20:15:17.947182 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:17.947177    4484 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"
Oct 31 20:15:17.947597 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:17.947220    4484 scope.go:122] "RemoveContainer" containerID="977022e2656c6532fa3c836a1692c7ee2ed222e95d5292e9d3714873642d6b67"
Oct 31 20:15:18.124755 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:18.124719    4484 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"]
Oct 31 20:15:18.175298 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:18.175262    4484 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891295-fs7vr"]
Oct 31 20:15:19.992404 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:19.992371    4484 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42c75fd-c612-494f-9238-41fce8bbf059" path="/var/lib/kubelet/pods/b42c75fd-c612-494f-9238-41fce8bbf059/volumes"
Oct 31 20:15:25.530961 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:15:25.530916    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:15:36.327731 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:36.327639    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fs5lm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:15:36.328532 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:36.328446    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fs5lm" podUID="8082f048-2aba-4b78-8f8e-dedf3015ea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:15:47.454670 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:47.454622    4484 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-g4xdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:15:47.455204 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:47.454715    4484 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-g4xdk" podUID="a7dcf339-7a9f-47df-acef-a2ab94742e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:15:55.159612 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:55.159462    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 20:15:55.172429 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:55.172401    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vsrqn_0a79a203-4e31-41e5-af54-c8d07320badd/ovn-acl-logging/0.log"
Oct 31 20:15:55.468899 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:15:55.468812    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:15:56.744055 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:56.744020    4484 scope.go:122] "RemoveContainer" containerID="acf5cd64f460ed370daceb24c25d92564f2f07f952c60d8fa53ef02b30dea396"
Oct 31 20:15:56.755444 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:15:56.754264    4484 scope.go:122] "RemoveContainer" containerID="02afe16c4c26d0e901de3a786401573c1bce1b6a60cad380cbc63293ae18ced8"
Oct 31 20:16:06.293603 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:06.293559    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-z57jb"]
Oct 31 20:16:25.469588 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:16:25.469549    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:16:37.515166 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:37.515133    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c8b7d8cc9-rzk2k_ea33ee05-e391-4d64-a57a-039b32d83cd2/manager/1.log"
Oct 31 20:16:37.884488 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:37.884370    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hf75l_30e22b48-ac4a-4c9c-8154-79b160d5c529/manager/0.log"
Oct 31 20:16:39.220088 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:39.220051    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64fbfd7b7f-5rqsl_f2bdb1a0-9e75-487d-a766-1c7bd313e7c4/manager/0.log"
Oct 31 20:16:39.544703 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:39.544669    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vhjgq_26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8/server/0.log"
Oct 31 20:16:40.135965 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:40.135919    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xxg6_9fc7ddc9-d304-433d-9f6a-55d677ec5ad0/proxy/0.log"
Oct 31 20:16:40.782819 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:40.782787    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c8b7d8cc9-rzk2k_ea33ee05-e391-4d64-a57a-039b32d83cd2/manager/0.log"
Oct 31 20:16:41.744796 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:41.744765    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68474f84db-lnkqx_a7d65473-dd86-4929-8f15-9239f45f33db/maas-api/0.log"
Oct 31 20:16:45.655667 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:45.655630    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-m59v4_916306e5-63fe-465c-85bf-8957717253de/postgres/0.log"
Oct 31 20:16:49.229916 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:49.229885    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-z57jb_5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a/limitador/0.log"
Oct 31 20:16:51.487228 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:51.487199    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d4f97d94f-8v994_a70aa0a3-26be-42df-881e-2d9f6a58697d/kube-auth-proxy/0.log"
Oct 31 20:16:52.689928 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:52.689887    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-fs5lm_8082f048-2aba-4b78-8f8e-dedf3015ea89/istio-proxy/0.log"
Oct 31 20:16:53.019458 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:53.019422    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-g4xdk_a7dcf339-7a9f-47df-acef-a2ab94742e15/istio-proxy/0.log"
Oct 31 20:16:53.711793 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:53.711758    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-vnvmf_a9f80747-2277-4637-a04a-15207891a808/istio-proxy/0.log"
Oct 31 20:16:54.486664 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:54.486631    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-86cf587c44-nbkn8_7856c13c-9c85-4d24-9149-55953e517e57/payload-pre-processing/0.log"
Oct 31 20:16:54.712847 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:54.712817    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-596fc796b7-8thbk_0b48a0e7-657c-46cf-9818-6bb3428fc94c/payload-processing/0.log"
Oct 31 20:16:55.469181 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:16:55.469144    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:16:56.022572 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:56.022537    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-7knwj_e86af09b-1972-4b3a-9c2e-7ba2b8b83670/main/0.log"
Oct 31 20:16:56.294777 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:56.294700    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-lwj8x_6683cf79-6892-4279-b3ad-65a6229a4a18/main/0.log"
Oct 31 20:16:56.831023 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:56.830990    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-8k7kf_7dc3fdbe-f1d9-4d47-82d9-e46f3366d5d2/main/0.log"
Oct 31 20:16:57.572513 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:16:57.572487    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-nx9hd_c1e45f78-79e3-4e67-a2c2-0ff4a291374b/main/0.log"
Oct 31 20:17:16.835862 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:16.835769    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h72m5_c4d62e4b-14f2-4ae5-83e4-bddf1230ddd7/global-pull-secret-syncer/0.log"
Oct 31 20:17:20.014358 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:20.014317    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-82t6b_e0b37043-a08e-475d-b8ff-3c8c2c8b653f/konnectivity-agent/0.log"
Oct 31 20:17:22.222638 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:22.222604    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-120.us-west-1.compute.internal_eb7aca0bcf277ba7408a15c0b72c2378/haproxy/0.log"
Oct 31 20:17:25.466535 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:25.466502    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c8b7d8cc9-rzk2k_ea33ee05-e391-4d64-a57a-039b32d83cd2/manager/0.log"
Oct 31 20:17:25.471370 ip-10-0-142-120 kubenswrapper[4484]: E1031 20:17:25.471338    4484 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gwg8k" containerName="extractor"
Oct 31 20:17:25.485466 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:25.485433    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c8b7d8cc9-rzk2k_ea33ee05-e391-4d64-a57a-039b32d83cd2/manager/1.log"
Oct 31 20:17:25.820214 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:25.820181    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hf75l_30e22b48-ac4a-4c9c-8154-79b160d5c529/manager/0.log"
Oct 31 20:17:26.765029 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.764986    4484 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"]
Oct 31 20:17:26.765634 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.765602    4484 cpu_manager.go:414] "containerMap: removing container" podUID="b42c75fd-c612-494f-9238-41fce8bbf059" containerName="cleanup"
Oct 31 20:17:26.765634 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.765635    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42c75fd-c612-494f-9238-41fce8bbf059" containerName="cleanup"
Oct 31 20:17:26.765823 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.765666    4484 cpu_manager.go:414] "containerMap: removing container" podUID="b42c75fd-c612-494f-9238-41fce8bbf059" containerName="cleanup"
Oct 31 20:17:26.765823 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.765671    4484 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42c75fd-c612-494f-9238-41fce8bbf059" containerName="cleanup"
Oct 31 20:17:26.765823 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.765758    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42c75fd-c612-494f-9238-41fce8bbf059" containerName="cleanup"
Oct 31 20:17:26.765823 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.765775    4484 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42c75fd-c612-494f-9238-41fce8bbf059" containerName="cleanup"
Oct 31 20:17:26.769114 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.769089    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.824410 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.824367    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bxfnk\"/\"openshift-service-ca.crt\""
Oct 31 20:17:26.850839 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.850797    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dc350ea-758f-4c95-bf95-5a654f5b9b15-sys\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.851073 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.850943    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv2l7\" (UniqueName: \"kubernetes.io/projected/7dc350ea-758f-4c95-bf95-5a654f5b9b15-kube-api-access-hv2l7\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.851073 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.850997    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7dc350ea-758f-4c95-bf95-5a654f5b9b15-podres\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.851073 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.851025    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7dc350ea-758f-4c95-bf95-5a654f5b9b15-proc\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.851268 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.851107    4484 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dc350ea-758f-4c95-bf95-5a654f5b9b15-lib-modules\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.874939 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.874903    4484 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bxfnk\"/\"kube-root-ca.crt\""
Oct 31 20:17:26.875120 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.875075    4484 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bxfnk\"/\"default-dockercfg-l2c59\""
Oct 31 20:17:26.877064 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.877040    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"]
Oct 31 20:17:26.951720 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.951681    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dc350ea-758f-4c95-bf95-5a654f5b9b15-sys\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.951923 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.951777    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv2l7\" (UniqueName: \"kubernetes.io/projected/7dc350ea-758f-4c95-bf95-5a654f5b9b15-kube-api-access-hv2l7\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.951923 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.951841    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7dc350ea-758f-4c95-bf95-5a654f5b9b15-podres\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.951923 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.951877    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7dc350ea-758f-4c95-bf95-5a654f5b9b15-proc\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.951923 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.951875    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dc350ea-758f-4c95-bf95-5a654f5b9b15-sys\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.951923 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.951899    4484 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dc350ea-758f-4c95-bf95-5a654f5b9b15-lib-modules\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.952130 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.951947    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7dc350ea-758f-4c95-bf95-5a654f5b9b15-podres\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.952130 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.951962    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7dc350ea-758f-4c95-bf95-5a654f5b9b15-proc\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.952130 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.951972    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dc350ea-758f-4c95-bf95-5a654f5b9b15-lib-modules\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:26.987048 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:26.987010    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64fbfd7b7f-5rqsl_f2bdb1a0-9e75-487d-a766-1c7bd313e7c4/manager/0.log"
Oct 31 20:17:27.010251 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:27.010219    4484 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv2l7\" (UniqueName: \"kubernetes.io/projected/7dc350ea-758f-4c95-bf95-5a654f5b9b15-kube-api-access-hv2l7\") pod \"perf-node-gather-daemonset-rvddm\" (UID: \"7dc350ea-758f-4c95-bf95-5a654f5b9b15\") " pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:27.085596 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:27.085451    4484 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:27.236337 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:27.236306    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vhjgq_26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8/server/0.log"
Oct 31 20:17:27.467106 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:27.467076    4484 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"]
Oct 31 20:17:27.468762 ip-10-0-142-120 kubenswrapper[4484]: W1031 20:17:27.468731    4484 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7dc350ea_758f_4c95_bf95_5a654f5b9b15.slice/crio-ae4eed9ff64984451ccd77fdc20c907704192fcf7cc72563124d8b0e2a366a3c WatchSource:0}: Error finding container ae4eed9ff64984451ccd77fdc20c907704192fcf7cc72563124d8b0e2a366a3c: Status 404 returned error can't find the container with id ae4eed9ff64984451ccd77fdc20c907704192fcf7cc72563124d8b0e2a366a3c
Oct 31 20:17:27.718649 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:27.718542    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm" event={"ID":"7dc350ea-758f-4c95-bf95-5a654f5b9b15","Type":"ContainerStarted","Data":"6db70d384604953bfc2e4fc9ee36fcc553830ad513187064465a300763954435"}
Oct 31 20:17:27.718649 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:27.718603    4484 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm" event={"ID":"7dc350ea-758f-4c95-bf95-5a654f5b9b15","Type":"ContainerStarted","Data":"ae4eed9ff64984451ccd77fdc20c907704192fcf7cc72563124d8b0e2a366a3c"}
Oct 31 20:17:27.718904 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:27.718731    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:28.542394 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:28.542365    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-z57jb_5eb6bbc4-2e5f-4b4d-9146-70c70cb8891a/limitador/0.log"
Oct 31 20:17:28.605962 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:28.605925    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xxg6_9fc7ddc9-d304-433d-9f6a-55d677ec5ad0/proxy/0.log"
Oct 31 20:17:31.504013 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:31.503960    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c8b7d8cc9-rzk2k_ea33ee05-e391-4d64-a57a-039b32d83cd2/manager/0.log"
Oct 31 20:17:31.522227 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:31.522181    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c8b7d8cc9-rzk2k_ea33ee05-e391-4d64-a57a-039b32d83cd2/manager/1.log"
Oct 31 20:17:31.856839 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:31.856749    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hf75l_30e22b48-ac4a-4c9c-8154-79b160d5c529/manager/0.log"
Oct 31 20:17:32.853830 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:32.853796    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64fbfd7b7f-5rqsl_f2bdb1a0-9e75-487d-a766-1c7bd313e7c4/manager/0.log"
Oct 31 20:17:33.025366 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:33.025331    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vhjgq_26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8/server/0.log"
Oct 31 20:17:33.735115 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:33.735071    4484 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm"
Oct 31 20:17:33.853316 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:33.853255    4484 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bxfnk/perf-node-gather-daemonset-rvddm" podStartSLOduration=7.853234091 podStartE2EDuration="7.853234091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 20:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:17:27.839599811 +0000 UTC m=+6993.986049668" watchObservedRunningTime="2026-10-31 20:17:33.853234091 +0000 UTC m=+6999.999683951"
Oct 31 20:17:34.394893 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:34.394861    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xxg6_9fc7ddc9-d304-433d-9f6a-55d677ec5ad0/proxy/0.log"
Oct 31 20:17:36.669958 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:36.669927    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-nj5kx_56dbaf32-7147-4622-afef-37fb8375bd4f/monitoring-plugin/0.log"
Oct 31 20:17:37.093038 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:37.093003    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f8cn5_91cf1206-44e3-4b7b-84a7-f6be0821bab4/node-exporter/0.log"
Oct 31 20:17:37.304632 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:37.304593    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c8b7d8cc9-rzk2k_ea33ee05-e391-4d64-a57a-039b32d83cd2/manager/0.log"
Oct 31 20:17:37.321914 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:37.321881    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-c8b7d8cc9-rzk2k_ea33ee05-e391-4d64-a57a-039b32d83cd2/manager/1.log"
Oct 31 20:17:37.557939 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:37.557915    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f8cn5_91cf1206-44e3-4b7b-84a7-f6be0821bab4/kube-rbac-proxy/0.log"
Oct 31 20:17:37.573597 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:37.573566    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hf75l_30e22b48-ac4a-4c9c-8154-79b160d5c529/manager/0.log"
Oct 31 20:17:37.973164 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:37.973091    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f8cn5_91cf1206-44e3-4b7b-84a7-f6be0821bab4/init-textfile/0.log"
Oct 31 20:17:38.596348 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:38.596312    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64fbfd7b7f-5rqsl_f2bdb1a0-9e75-487d-a766-1c7bd313e7c4/manager/0.log"
Oct 31 20:17:38.753628 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:38.753597    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-vhjgq_26ab5dc0-00d0-4ce9-a2ef-207ad132bfb8/server/0.log"
Oct 31 20:17:40.195494 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:40.195444    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-6xxg6_9fc7ddc9-d304-433d-9f6a-55d677ec5ad0/proxy/0.log"
Oct 31 20:17:42.163075 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:42.163046    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_57064eeb-d684-49f8-8eec-f85bf72412d0/prometheus/0.log"
Oct 31 20:17:42.574829 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:42.574794    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_57064eeb-d684-49f8-8eec-f85bf72412d0/config-reloader/0.log"
Oct 31 20:17:42.988208 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:42.988180    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_57064eeb-d684-49f8-8eec-f85bf72412d0/thanos-sidecar/0.log"
Oct 31 20:17:43.402342 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:43.402258    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_57064eeb-d684-49f8-8eec-f85bf72412d0/kube-rbac-proxy-web/0.log"
Oct 31 20:17:43.821538 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:43.821507    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_57064eeb-d684-49f8-8eec-f85bf72412d0/kube-rbac-proxy/0.log"
Oct 31 20:17:44.233937 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:44.233845    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_57064eeb-d684-49f8-8eec-f85bf72412d0/kube-rbac-proxy-thanos/0.log"
Oct 31 20:17:44.649928 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:44.649852    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_57064eeb-d684-49f8-8eec-f85bf72412d0/init-config-reloader/0.log"
Oct 31 20:17:45.489729 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:45.489697    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d4f97d94f-8v994_a70aa0a3-26be-42df-881e-2d9f6a58697d/kube-auth-proxy/0.log"
Oct 31 20:17:45.944086 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:45.944005    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-bww7c_d9298c7e-36d1-4ee0-862e-d3d875b8c1f6/prometheus-operator-admission-webhook/0.log"
Oct 31 20:17:46.574843 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:46.574814    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-fs5lm_8082f048-2aba-4b78-8f8e-dedf3015ea89/istio-proxy/0.log"
Oct 31 20:17:46.788190 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:46.788158    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-g4xdk_a7dcf339-7a9f-47df-acef-a2ab94742e15/istio-proxy/0.log"
Oct 31 20:17:47.427040 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:47.427005    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-vnvmf_a9f80747-2277-4637-a04a-15207891a808/istio-proxy/0.log"
Oct 31 20:17:48.056270 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:48.056236    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-86cf587c44-nbkn8_7856c13c-9c85-4d24-9149-55953e517e57/payload-pre-processing/0.log"
Oct 31 20:17:48.267090 ip-10-0-142-120 kubenswrapper[4484]: I1031 20:17:48.267064    4484 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-596fc796b7-8thbk_0b48a0e7-657c-46cf-9818-6bb3428fc94c/payload-processing/0.log"